Sep 16 12:47:51.941179 ip-10-0-143-99 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:47:52.362431 ip-10-0-143-99 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:52.362431 ip-10-0-143-99 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:47:52.362431 ip-10-0-143-99 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:52.362431 ip-10-0-143-99 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:47:52.362431 ip-10-0-143-99 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:52.363409 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.363331    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:47:52.368373 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368358    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:52.368373 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368372    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368376    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368379    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368382    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368384    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368387    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368390    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368392    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368394    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368397    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368399    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368402    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368404    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368407    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368414    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368417    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368419    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368422    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368424    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368427    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:52.368442 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368429    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368431    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368434    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368436    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368439    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368441    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368443    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368447    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368450    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368452    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368455    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368458    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368460    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368463    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368466    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368468    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368470    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368472    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368475    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368477    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:52.368931 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368479    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368482    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368486    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368489    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368493    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368495    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368498    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368500    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368503    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368506    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368508    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368510    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368513    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368516    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368520    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368524    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368527    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368529    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368532    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:52.369416 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368534    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368536    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368539    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368541    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368544    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368547    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368550    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368552    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368555    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368557    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368560    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368562    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368566    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368568    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368571    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368573    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368575    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368578    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368580    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368583    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:52.369893 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368585    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368588    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368590    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368594    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368597    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368600    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368603    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368606    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368610    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368613    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368616    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368619    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368622    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368624    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368627    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368630    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368632    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368635    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.368637    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:52.370410 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369313    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369319    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369322    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369325    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369328    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369331    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369333    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369336    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369339    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369341    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369344    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369346    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369348    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369351    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369353    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369356    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369358    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369361    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369363    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369366    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:52.370903 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369369    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369372    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369374    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369377    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369379    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369382    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369384    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369387    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369389    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369391    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369394    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369396    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369399    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369401    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369404    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369406    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369408    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369411    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369417    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369420    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:52.371397 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369422    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369425    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369428    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369430    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369433    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369435    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369437    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369440    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369442    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369445    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369447    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369451    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369455    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369459    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369462    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369467    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369471    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369474    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369477    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:52.371901 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369480    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369482    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369485    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369487    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369490    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369492    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369495    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369497    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369501    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369503    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369506    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369508    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369511    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369514    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369517    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369521    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369524    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369527    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369529    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:52.372357 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369532    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369534    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369537    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369540    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369543    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369546    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369549    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369552    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369556    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369559    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369561    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369564    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369566    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369569    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369571    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369574    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369576    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369579    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369581    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:52.372826 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369584    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369586    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369954    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369959    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369962    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369964    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369967    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369969    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369972    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369975    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369977    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369980    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369983    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369985    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369988    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369990    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369993    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369996    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.369998    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370001    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:52.373295 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370004    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370006    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370009    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370011    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370014    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370018    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370020    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370022    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370025    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370027    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370030    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370032    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370034    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370037    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370039    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370042    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370044    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370047    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370049    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370052    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:52.373790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370055    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370058    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370060    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370063    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370066    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370068    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370070    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370073    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370075    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370079    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370082    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370085    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370090    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370093    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370096    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370099    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370101    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370104    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370107    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370109    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:52.374383 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370111    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370114    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370116    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370119    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370121    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370123    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370126    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370129    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370131    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370134    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370136    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370139    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370142    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370145    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370147    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370150    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370152    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370155    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370157    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:52.374876 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370159    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370162    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370164    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370166    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370169    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370171    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370173    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370176    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370178    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370181    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370183    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370186    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370188    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370191    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370193    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370196    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370198    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370202    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370205    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:52.375335 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370208    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370210    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370213    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370287    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370321    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370328    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370332    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370337    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370340    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370344    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370348    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370351    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370353    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370357    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370360    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370362    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370367    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370370    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370373    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370376    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370379    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370383    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370385    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:47:52.375879 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370388    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370391    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370394    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370397    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370400    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370403    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370406    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370409    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370412    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370417    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370420    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370423    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370427    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370430    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370433    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370435    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370438    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370444    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370447    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370450    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370453    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370456    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370458    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370462    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:47:52.376459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370465    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370468    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370471    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370474    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370476    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370479    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370482    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370484    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370487    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370490    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370493    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370496    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370499    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370502    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370504    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370507    2577 flags.go:64] FLAG: --help="false"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370510    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370513    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370515    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370518    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370522    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370525    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370528    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:47:52.377052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370530    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370533    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370536    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370539    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370542    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370545    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370547    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370550    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370553    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370556    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370559    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370562    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370564    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370567    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370573    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370575    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370578    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370581    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370584    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370587    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370590    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370592    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370596    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370599    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370603    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:47:52.377626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370606    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370608    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370611    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370614    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370617    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370620    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370622    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370629    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370632    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370634    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370637    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370641    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370646    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370649    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370652    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370655    2577 flags.go:64] FLAG: --port="10250"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370658    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370661    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06bdafc428aca1604"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370664    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370667    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370670    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370673    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370676    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370679    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:47:52.378252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370681    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370684    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370688    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370690    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370693    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370696    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370699    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370701    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370704    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370707    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370710    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370713    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370715    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370718    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370721    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370724    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370726    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370729    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370732    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370734    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370737    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370740    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370745    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370748    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370751    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:47:52.378852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370754    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370757    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370759    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370762    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370764    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370767    2577 flags.go:64] FLAG: --v="2"
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370771    2577 flags.go:64] FLAG: --version="false"
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370774    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370777    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.370780    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370911    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370915    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370919    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370921    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370924    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370926    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370929    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370931    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370936    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370938    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370941    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370945    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:52.379514 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370948    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370952    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370954    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370957    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370960    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370963    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370966    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370968    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370970    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370974    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370976    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370979    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370982    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370984    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370987    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370989    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370992    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370994    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370997    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:52.380051 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.370999    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371002    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371004    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371006    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371009    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371011    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371014    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371016    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371019    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371022    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371025    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371027    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371030    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371033    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371035    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371037    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371040    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371042    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371045    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371047    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:52.380511 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371050    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371053    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371057    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371059    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371062    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371065    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371069    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371071    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371074    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371076    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371079    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371082    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371085    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371088    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371091    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371094    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371097    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371099    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371102    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:52.381004 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371104    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371107    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371111    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371113    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371116    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371118    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371121    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371123    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371126    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371128    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371131    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371133    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371136    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371138    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371142    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371145    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371148    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371150    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371152    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371155    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:52.381469 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371157    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371160    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371163    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371165    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371168    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371171    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371173    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371176    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.371178    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.372346    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380076    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380091    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380104    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380109    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380163    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:52.381978 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380167    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380171    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380173    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380176    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380179    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380181    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380184    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380187    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380189    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380192    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380194    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380197    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380199    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380202    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380205    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380207    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380210    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380212    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380214    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380217    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:52.382339 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380220    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380223    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380227    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380230    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380233    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380236    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380238    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380241    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380245    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380248    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380251    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380253    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380255    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380258    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380261    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380263    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380266    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380268    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380271    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:52.382848 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380274    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380276    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380278    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380281    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380283    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380286    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380288    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380291    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380293    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380296    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380298    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380300    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380303    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380305    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380308    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380312    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380315    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380318    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380321    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:52.383320 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380324    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380326    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380329    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380332    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380335    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380338    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380340    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380343    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380345    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380347    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380350    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380352    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380354    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380357    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380360    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380362    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380364    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380367    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380369    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380371    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:52.383778 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380374    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380376    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380379    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380381    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380384    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380386    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380389    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380391    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380393    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380396    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380400    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380403    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380406    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380408    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380412    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380414    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380417    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380419    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380422    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:52.384345 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380424    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380429    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380534    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380539    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380542    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380545    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380548    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380550    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380553    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380556    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380559    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380562    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380565    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380569    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380572    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:52.384810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380575    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380578    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380581    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380583    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380586    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380588    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380590    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380593    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380595    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380597    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380600    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380603    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380606    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380609    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380611    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380614    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380617    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380619    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380621    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380624    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:52.385195 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380626    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380629    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380631    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380634    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380637    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380639    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380641    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380644    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380646    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380648    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380651    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380653    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380655    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380658    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380660    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380663    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380667    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380670    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380672    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380675    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:52.385688 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380678    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380680    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380683    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380686    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380689    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380692    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380695    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380697    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380701    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380703    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380706    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380709    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380712    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380714    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380717    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380719    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380722    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380724    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380727    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380729    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:52.386203 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380732    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380734    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380736    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380739    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380741    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380744    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380746    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380749    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380751    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380754    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380756    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380759    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380763    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380765    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380768    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380770    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380773    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380776    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380778    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:52.386685 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380780    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380783    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380786    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380788    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380790    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380793    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:52.380795    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380800    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.380910    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.383222    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.384231    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.384330    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:52.387218 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.384367    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:52.388967 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.388937    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:52.390157 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.390138    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:52.401507 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.401489    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:47:52.405055 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.405039    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 12:47:52.406103 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.406082    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:47:52.409434 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.409410    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e7ce281b-9544-40de-bb90-ed23df9e8e65:/dev/nvme0n1p4 f3935419-841c-4063-a792-b557b120cb22:/dev/nvme0n1p3]
Sep 16 12:47:52.409505 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.409433    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:47:52.414418 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.414309    2577 manager.go:216] Machine: {Timestamp:2026-09-16 12:47:52.412922945 +0000 UTC m=+0.363268490 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100437 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26115d6c6ef46f8b9861d052b55f69 SystemUUID:ec26115d-6c6e-f46f-8b98-61d052b55f69 BootID:06de9dbd-f32b-472c-b501-e738355b6db4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:ad:39:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:ad:39:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:8f:55:fc:b9:ea Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:47:52.414418 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.414413    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:47:52.414516 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.414484    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:47:52.416439 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.416424    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:52.418077 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.418056    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:47:52.418210 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.418081    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:47:52.419114 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.419104    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:47:52.419144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.419117    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:47:52.419144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.419125    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:52.419906 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.419896    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:52.421058 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.421048    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:47:52.426469 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426457    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:52.426580 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426570    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:47:52.426609 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426602    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:47:52.426644 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426614    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:47:52.426644 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426628    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:47:52.426644 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426637    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:47:52.426725 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426646    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:47:52.426987 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.426976    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:52.427030 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.427000    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:52.431877 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.431835    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:47:52.434672 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.434657    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:47:52.434744 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.434688    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:47:52.436083 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.436061    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:47:52.436178 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.436161    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:47:52.436668 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436645    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:47:52.436668 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436662    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:47:52.436668 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436668    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436673    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436684    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436691    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436703    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436709    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436715    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436720    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436728    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:47:52.436783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.436733    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:47:52.437612 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.437602    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:47:52.437612 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.437612    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:47:52.440228 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.440213    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2z8sb"
Sep 16 12:47:52.441177 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.441166    2577 server.go:1268] "Started kubelet"
Sep 16 12:47:52.441278 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.441249    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:47:52.441339 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.441257    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:47:52.441339 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.441323    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:47:52.442030 ip-10-0-143-99 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:47:52.442909 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.442830    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:47:52.443374 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.443361    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:47:52.447644 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.447629    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2z8sb"
Sep 16 12:47:52.448618 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.448602    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:47:52.449045 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.449032    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:47:52.449993 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.448928    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-99.ec2.internal.18d5cdf37314f7cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-99.ec2.internal,UID:ip-10-0-143-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-99.ec2.internal,},FirstTimestamp:2026-09-16 12:47:52.441141195 +0000 UTC m=+0.391486740,LastTimestamp:2026-09-16 12:47:52.441141195 +0000 UTC m=+0.391486740,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-99.ec2.internal,}"
Sep 16 12:47:52.451093 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.451080    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:47:52.451093 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.451086    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:52.451201 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.451153    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.452012 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.451995    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:47:52.452012 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.451996    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:47:52.452155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452020    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:47:52.452155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452049    2577 factory.go:55] Registering systemd factory
Sep 16 12:47:52.452155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452057    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:47:52.452155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452023    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:47:52.452155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452151    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:47:52.452155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452156    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:47:52.452330 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452294    2577 factory.go:153] Registering CRI-O factory
Sep 16 12:47:52.452330 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452304    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:47:52.452330 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452319    2577 factory.go:103] Registering Raw factory
Sep 16 12:47:52.452330 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452330    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:47:52.452644 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.452632    2577 manager.go:349] Starting recovery of all containers
Sep 16 12:47:52.453988 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.453960    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:47:52.454095 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.454072    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:47:52.463385 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.463311    2577 manager.go:354] Recovery completed
Sep 16 12:47:52.465218 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.465193    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 12:47:52.468899 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.468887    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.470536 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.470522    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.470607 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.470554    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.470607 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.470569    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.471087 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471061    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:47:52.471087 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471077    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:47:52.471178 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471095    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:52.471178 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471096    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.471692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471680    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.471743 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471706    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.471743 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.471719    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.473058 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.473046    2577 policy_none.go:49] "None policy: Start"
Sep 16 12:47:52.473093 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.473062    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:47:52.473093 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.473073    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:47:52.475029 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.475019    2577 policy_none.go:47] "Start"
Sep 16 12:47:52.516522 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.516509    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.516540    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.516549    2577 server.go:87] "Starting device plugin registration server"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.516998    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.517012    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.517151    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.517250    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.517275    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.517676    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:47:52.540692 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.517725    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.565118 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.565098    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:47:52.566388 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.566372    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:47:52.566458 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.566417    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:47:52.566458 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.566438    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:47:52.566539 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.566477    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:47:52.568368 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.568350    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:52.617524 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.617470    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.622180 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.622166    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.622257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.622192    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.622257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.622201    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.622257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.622234    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.629400 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.629383    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.629494 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.629405    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-99.ec2.internal\": node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.652816 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.652795    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.666835 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.666802    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"]
Sep 16 12:47:52.666927 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.666882    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.667683 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.667670    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.667755 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.667700    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.667755 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.667714    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.670223 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.670208    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.670325 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.670312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.670373 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.670337    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.670990 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.670975    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.670990 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.670989    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.671114 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.671002    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.671114 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.671012    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.671114 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.671027    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.671114 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.671015    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.673209 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.673195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.673270 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.673218    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:52.675392 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.675373    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:52.675481 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.675405    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:52.675481 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.675417    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:52.689373 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.689354    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-99.ec2.internal\" not found" node="ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.694489 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.694473    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-99.ec2.internal\" not found" node="ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.753824 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.753801    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.853043 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.853011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562cd238c1a07a23034785db4cb30206-config\") pod \"kube-apiserver-proxy-ip-10-0-143-99.ec2.internal\" (UID: \"562cd238c1a07a23034785db4cb30206\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.853043 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.853045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6aeab2e70341bb2c67f5334fcea3a0e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal\" (UID: \"6aeab2e70341bb2c67f5334fcea3a0e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.853160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.853071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aeab2e70341bb2c67f5334fcea3a0e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal\" (UID: \"6aeab2e70341bb2c67f5334fcea3a0e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.854077 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.854063    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.953663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.953645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562cd238c1a07a23034785db4cb30206-config\") pod \"kube-apiserver-proxy-ip-10-0-143-99.ec2.internal\" (UID: \"562cd238c1a07a23034785db4cb30206\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.953733 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.953664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6aeab2e70341bb2c67f5334fcea3a0e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal\" (UID: \"6aeab2e70341bb2c67f5334fcea3a0e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.953733 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.953679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aeab2e70341bb2c67f5334fcea3a0e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal\" (UID: \"6aeab2e70341bb2c67f5334fcea3a0e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.953733 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.953702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aeab2e70341bb2c67f5334fcea3a0e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal\" (UID: \"6aeab2e70341bb2c67f5334fcea3a0e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.953733 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.953732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6aeab2e70341bb2c67f5334fcea3a0e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal\" (UID: \"6aeab2e70341bb2c67f5334fcea3a0e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.953910 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.953754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562cd238c1a07a23034785db4cb30206-config\") pod \"kube-apiserver-proxy-ip-10-0-143-99.ec2.internal\" (UID: \"562cd238c1a07a23034785db4cb30206\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.954708 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:52.954694    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:52.991889 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.991869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:52.996464 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:52.996398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:53.055275 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.055237    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.155725 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.155698    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.256212 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.256162    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.356733 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.356709    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.383894 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:53.383873    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:47:53.384282 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:53.384039    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:47:53.454210 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:53.454180    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:42:52 +0000 UTC" deadline="2028-05-13 02:33:27.168704809 +0000 UTC"
Sep 16 12:47:53.454210 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:53.454206    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14509h45m33.714503648s"
Sep 16 12:47:53.457652 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.457634    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.558564 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.558501    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.567352 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:53.567328    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:53.589054 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:53.589030    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562cd238c1a07a23034785db4cb30206.slice/crio-8bb773a443972e7fe3d9ce853bf14700ddd60e9933bf1486e6f678ba903ddada WatchSource:0}: Error finding container 8bb773a443972e7fe3d9ce853bf14700ddd60e9933bf1486e6f678ba903ddada: Status 404 returned error can't find the container with id 8bb773a443972e7fe3d9ce853bf14700ddd60e9933bf1486e6f678ba903ddada
Sep 16 12:47:53.589455 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:53.589437    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeab2e70341bb2c67f5334fcea3a0e2.slice/crio-bc72f718346a48e23337eff3b870a013f50260231f477bf42a131d7f3886738f WatchSource:0}: Error finding container bc72f718346a48e23337eff3b870a013f50260231f477bf42a131d7f3886738f: Status 404 returned error can't find the container with id bc72f718346a48e23337eff3b870a013f50260231f477bf42a131d7f3886738f
Sep 16 12:47:53.593996 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:53.593974    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:47:53.659314 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.659291    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.759762 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.759740    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.860240 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.860193    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:53.960994 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:53.960963    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-99.ec2.internal\" not found"
Sep 16 12:47:54.006270 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.006240    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:54.028910 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.028881    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:54.052291 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.052265    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:54.065153 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.065129    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:47:54.066291 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.066274    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal"
Sep 16 12:47:54.082971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.082942    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:47:54.299263 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.299228    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:54.427678 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.427603    2577 apiserver.go:52] "Watching apiserver"
Sep 16 12:47:54.435601 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.435559    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:47:54.436938 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.436910    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-p6wfm","openshift-image-registry/node-ca-8jkzg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal","openshift-multus/multus-jt5qw","openshift-multus/network-metrics-daemon-mww2q","openshift-ovn-kubernetes/ovnkube-node-q8psw","kube-system/global-pull-secret-syncer-bj87h","kube-system/konnectivity-agent-xnvx4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2","openshift-multus/multus-additional-cni-plugins-xmcvr","openshift-network-diagnostics/network-check-target-fr8w4","openshift-network-operator/iptables-alerter-4h482","kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q46rc"]
Sep 16 12:47:54.440343 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.439925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.440343 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.440009    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:47:54.442676 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.442657    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:54.442761 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.442730    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:47:54.447624 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.447602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.449840 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.449813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.450810 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.450787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-djt86\""
Sep 16 12:47:54.453468 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.452939    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:54.453468 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.453079    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.453468 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.453245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.453468 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.453286    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:47:54.454220 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.453802    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.454220 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.454164    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:47:54.454397 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.454368    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.454613 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.454598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:54.454706 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.454688    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:47:54.455103 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.455078    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vvmw6\""
Sep 16 12:47:54.456632 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.456605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.459371 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.459338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.461843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15cbea84-3f63-497b-b7db-7e88193b54d6-kubelet-config\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.461884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysctl-conf\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.461912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25lf\" (UniqueName: \"kubernetes.io/projected/8bb342f5-ba85-4bea-a1a5-fa75f526c896-kube-api-access-n25lf\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.461959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.461985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-cni-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-daemon-config\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462056    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-host\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxvk\" (UniqueName: \"kubernetes.io/projected/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-kube-api-access-lvxvk\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-cnibin\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ce327f0-b212-4e77-a282-cd4e727bdc04-cni-binary-copy\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-kubelet\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glsjb\" (UniqueName: \"kubernetes.io/projected/9ce327f0-b212-4e77-a282-cd4e727bdc04-kube-api-access-glsjb\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysconfig\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-run\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.462592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-tmp\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-hostroot\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-kubernetes\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-tuned\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-system-cni-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-conf-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-multus-certs\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15cbea84-3f63-497b-b7db-7e88193b54d6-dbus\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysctl-d\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-systemd\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-sys\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-socket-dir-parent\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-k8s-cni-cncf-io\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-cni-bin\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462062    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-etc-kubernetes\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.462927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-modprobe-d\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.463384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.463052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-cni-multus\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.464088 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.463075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:54.464088 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.463094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-lib-modules\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.464088 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.463122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.464088 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.463134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-var-lib-kubelet\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.464088 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.463497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-netns\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.464533 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.464512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x4kjb\""
Sep 16 12:47:54.464646 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.464628    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:47:54.464718 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.464634    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.464763 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.464719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:47:54.464813 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.464766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:47:54.464919 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.464903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.465016 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.465002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zctbx\""
Sep 16 12:47:54.465061 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.465037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.466077 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.466062    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-94rvv\""
Sep 16 12:47:54.466247 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.466202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:47:54.466247 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.466234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:47:54.466909 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.466887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.469671 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.469404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.472725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.473025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.473281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j5k2j\""
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.473339    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-44z7w\""
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.473697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.473834    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:47:54.476240 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.474189    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:47:54.477160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.477130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.480766 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.480101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.480766 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.480199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jh7wb\""
Sep 16 12:47:54.481559 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.481433    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:54.482775 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.482755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.482874 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.482762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:47:54.483106 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.483044    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:47:54.483106 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.483069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bt87z\""
Sep 16 12:47:54.483349 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.483290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:47:54.510037 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.510017    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-268wd"
Sep 16 12:47:54.519298 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.519277    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-268wd"
Sep 16 12:47:54.553647 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.553627    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:47:54.563866 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.563834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysctl-conf\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.563971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.563897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-cni-netd\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.563971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.563928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-tmp-dir\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.563971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.563958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.564144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.563992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5759cfbb-d3e5-4129-b62e-00648ab43d8b-serviceca\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.564144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.564144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-registration-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.564144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-hosts-file\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.564144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-host\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.564144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxvk\" (UniqueName: \"kubernetes.io/projected/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-kube-api-access-lvxvk\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ce327f0-b212-4e77-a282-cd4e727bdc04-cni-binary-copy\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-kubelet\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-systemd-units\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-run-netns\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovnkube-config\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84px\" (UniqueName: \"kubernetes.io/projected/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-kube-api-access-r84px\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysconfig\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.564381 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-hostroot\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-system-cni-dir\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrm7m\" (UniqueName: \"kubernetes.io/projected/ff789be7-69c9-40bb-a8aa-42f34de68c8f-kube-api-access-mrm7m\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-kubernetes\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-tuned\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-system-cni-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/588f3d87-3ab5-4b39-bb0b-2636e8411bb9-agent-certs\") pod \"konnectivity-agent-xnvx4\" (UID: \"588f3d87-3ab5-4b39-bb0b-2636e8411bb9\") " pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-systemd\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysctl-d\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.564663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-socket-dir-parent\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-etc-kubernetes\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-modprobe-d\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5759cfbb-d3e5-4129-b62e-00648ab43d8b-host\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-kubelet\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-log-socket\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cnibin\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-var-lib-kubelet\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15cbea84-3f63-497b-b7db-7e88193b54d6-kubelet-config\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.564973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25lf\" (UniqueName: \"kubernetes.io/projected/8bb342f5-ba85-4bea-a1a5-fa75f526c896-kube-api-access-n25lf\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxbj\" (UniqueName: \"kubernetes.io/projected/5759cfbb-d3e5-4129-b62e-00648ab43d8b-kube-api-access-vfxbj\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-run-ovn-kubernetes\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-cni-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-daemon-config\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-cnibin\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glsjb\" (UniqueName: \"kubernetes.io/projected/9ce327f0-b212-4e77-a282-cd4e727bdc04-kube-api-access-glsjb\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-node-log\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-env-overrides\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pfz\" (UniqueName: \"kubernetes.io/projected/d9129bf1-95b9-47aa-83a7-2a76f3729b55-kube-api-access-h6pfz\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnk6\" (UniqueName: \"kubernetes.io/projected/3cbe3b34-d894-4097-b400-96f05b0d077b-kube-api-access-lpnk6\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxkq\" (UniqueName: \"kubernetes.io/projected/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-kube-api-access-fpxkq\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-iptables-alerter-script\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-run\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-tmp\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-etc-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.565483 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-conf-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-multus-certs\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15cbea84-3f63-497b-b7db-7e88193b54d6-dbus\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-var-lib-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-cni-bin\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-systemd\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-sys\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-k8s-cni-cncf-io\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-cni-bin\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/588f3d87-3ab5-4b39-bb0b-2636e8411bb9-konnectivity-ca\") pod \"konnectivity-agent-xnvx4\" (UID: \"588f3d87-3ab5-4b39-bb0b-2636e8411bb9\") " pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-slash\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovn-node-metrics-cert\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.565969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovnkube-script-lib\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-cni-multus\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-ovn\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-device-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-sys-fs\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-lib-modules\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-netns\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-socket-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-host-slash\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.566653 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.566534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysctl-conf\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.567159 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.566666    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:54.567159 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.566755    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:55.066724957 +0000 UTC m=+3.017070494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:54.567159 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.567054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-host\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.568193 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.568170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ce327f0-b212-4e77-a282-cd4e727bdc04-cni-binary-copy\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.568301 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.568263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-kubelet\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.568389 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.568373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysconfig\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.568445 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.568430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-hostroot\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.568621 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.568575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-kubernetes\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.569219 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.569195    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.569700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-system-cni-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.569848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-sysctl-d\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.569933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-socket-dir-parent\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.569974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-etc-kubernetes\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.570037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-modprobe-d\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.570131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-var-lib-kubelet\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.570176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15cbea84-3f63-497b-b7db-7e88193b54d6-kubelet-config\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.570588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-cni-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-daemon-config\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-cnibin\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-multus-conf-dir\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-multus-certs\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15cbea84-3f63-497b-b7db-7e88193b54d6-dbus\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-systemd\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-sys\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.571995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-k8s-cni-cncf-io\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.572040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-cni-bin\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.572173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-var-lib-cni-multus\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573060 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.572249    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:54.573926 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.572310    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:55.07229259 +0000 UTC m=+3.022638123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:54.573926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.572409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-lib-modules\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.572496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ce327f0-b212-4e77-a282-cd4e727bdc04-host-run-netns\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.573926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.572577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-run\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.573558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-etc-tuned\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.573926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.573642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-tmp\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.578403 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.578346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal" event={"ID":"562cd238c1a07a23034785db4cb30206","Type":"ContainerStarted","Data":"8bb773a443972e7fe3d9ce853bf14700ddd60e9933bf1486e6f678ba903ddada"}
Sep 16 12:47:54.578512 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.578417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal" event={"ID":"6aeab2e70341bb2c67f5334fcea3a0e2","Type":"ContainerStarted","Data":"bc72f718346a48e23337eff3b870a013f50260231f477bf42a131d7f3886738f"}
Sep 16 12:47:54.579534 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.579512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxvk\" (UniqueName: \"kubernetes.io/projected/3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e-kube-api-access-lvxvk\") pod \"tuned-q46rc\" (UID: \"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e\") " pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.581077 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.581051    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:54.581077 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.581075    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:54.581216 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.581089    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:54.581216 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:54.581140    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:47:55.081124131 +0000 UTC m=+3.031469664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:54.583987 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.583958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25lf\" (UniqueName: \"kubernetes.io/projected/8bb342f5-ba85-4bea-a1a5-fa75f526c896-kube-api-access-n25lf\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:54.584361 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.584334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glsjb\" (UniqueName: \"kubernetes.io/projected/9ce327f0-b212-4e77-a282-cd4e727bdc04-kube-api-access-glsjb\") pod \"multus-jt5qw\" (UID: \"9ce327f0-b212-4e77-a282-cd4e727bdc04\") " pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.667173 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5759cfbb-d3e5-4129-b62e-00648ab43d8b-host\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-kubelet\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-log-socket\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cnibin\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxbj\" (UniqueName: \"kubernetes.io/projected/5759cfbb-d3e5-4129-b62e-00648ab43d8b-kube-api-access-vfxbj\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5759cfbb-d3e5-4129-b62e-00648ab43d8b-host\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-kubelet\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-run-ovn-kubernetes\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-log-socket\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-node-log\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-env-overrides\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cnibin\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pfz\" (UniqueName: \"kubernetes.io/projected/d9129bf1-95b9-47aa-83a7-2a76f3729b55-kube-api-access-h6pfz\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-run-ovn-kubernetes\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnk6\" (UniqueName: \"kubernetes.io/projected/3cbe3b34-d894-4097-b400-96f05b0d077b-kube-api-access-lpnk6\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxkq\" (UniqueName: \"kubernetes.io/projected/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-kube-api-access-fpxkq\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-iptables-alerter-script\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-etc-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-var-lib-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-cni-bin\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/588f3d87-3ab5-4b39-bb0b-2636e8411bb9-konnectivity-ca\") pod \"konnectivity-agent-xnvx4\" (UID: \"588f3d87-3ab5-4b39-bb0b-2636e8411bb9\") " pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-slash\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.667765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovn-node-metrics-cert\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovnkube-script-lib\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-ovn\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-device-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-sys-fs\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-node-log\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-cni-bin\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-socket-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-host-slash\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-cni-netd\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-tmp-dir\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5759cfbb-d3e5-4129-b62e-00648ab43d8b-serviceca\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-registration-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.668690 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-hosts-file\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-systemd-units\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-run-netns\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovnkube-config\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84px\" (UniqueName: \"kubernetes.io/projected/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-kube-api-access-r84px\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-system-cni-dir\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.667999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-env-overrides\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrm7m\" (UniqueName: \"kubernetes.io/projected/ff789be7-69c9-40bb-a8aa-42f34de68c8f-kube-api-access-mrm7m\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/588f3d87-3ab5-4b39-bb0b-2636e8411bb9-agent-certs\") pod \"konnectivity-agent-xnvx4\" (UID: \"588f3d87-3ab5-4b39-bb0b-2636e8411bb9\") " pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-systemd\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-systemd\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-tmp-dir\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5759cfbb-d3e5-4129-b62e-00648ab43d8b-serviceca\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/588f3d87-3ab5-4b39-bb0b-2636e8411bb9-konnectivity-ca\") pod \"konnectivity-agent-xnvx4\" (UID: \"588f3d87-3ab5-4b39-bb0b-2636e8411bb9\") " pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-slash\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.669539 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-registration-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovnkube-script-lib\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.668939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-device-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-sys-fs\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-ovn\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-var-lib-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-iptables-alerter-script\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-etc-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-run-openvswitch\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cbe3b34-d894-4097-b400-96f05b0d077b-socket-dir\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-host-slash\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-cni-netd\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.670414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-hosts-file\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.671264 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-systemd-units\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.671264 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.669888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9129bf1-95b9-47aa-83a7-2a76f3729b55-host-run-netns\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.671264 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.670074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff789be7-69c9-40bb-a8aa-42f34de68c8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.671264 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.670136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff789be7-69c9-40bb-a8aa-42f34de68c8f-system-cni-dir\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.671264 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.670388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovnkube-config\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.672005 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.671967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9129bf1-95b9-47aa-83a7-2a76f3729b55-ovn-node-metrics-cert\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.673327 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.673298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/588f3d87-3ab5-4b39-bb0b-2636e8411bb9-agent-certs\") pod \"konnectivity-agent-xnvx4\" (UID: \"588f3d87-3ab5-4b39-bb0b-2636e8411bb9\") " pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.677507 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.676929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxbj\" (UniqueName: \"kubernetes.io/projected/5759cfbb-d3e5-4129-b62e-00648ab43d8b-kube-api-access-vfxbj\") pod \"node-ca-8jkzg\" (UID: \"5759cfbb-d3e5-4129-b62e-00648ab43d8b\") " pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.680765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.680712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84px\" (UniqueName: \"kubernetes.io/projected/06a8f2ab-0bc2-4dc7-b4bd-1acc18095840-kube-api-access-r84px\") pod \"iptables-alerter-4h482\" (UID: \"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840\") " pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:54.681334 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.681312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pfz\" (UniqueName: \"kubernetes.io/projected/d9129bf1-95b9-47aa-83a7-2a76f3729b55-kube-api-access-h6pfz\") pod \"ovnkube-node-q8psw\" (UID: \"d9129bf1-95b9-47aa-83a7-2a76f3729b55\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.681787 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.681761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxkq\" (UniqueName: \"kubernetes.io/projected/11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5-kube-api-access-fpxkq\") pod \"node-resolver-p6wfm\" (UID: \"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5\") " pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.681868 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.681845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnk6\" (UniqueName: \"kubernetes.io/projected/3cbe3b34-d894-4097-b400-96f05b0d077b-kube-api-access-lpnk6\") pod \"aws-ebs-csi-driver-node-8p8p2\" (UID: \"3cbe3b34-d894-4097-b400-96f05b0d077b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.682277 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.682252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrm7m\" (UniqueName: \"kubernetes.io/projected/ff789be7-69c9-40bb-a8aa-42f34de68c8f-kube-api-access-mrm7m\") pod \"multus-additional-cni-plugins-xmcvr\" (UID: \"ff789be7-69c9-40bb-a8aa-42f34de68c8f\") " pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.766920 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.766884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q46rc"
Sep 16 12:47:54.779145 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.779115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt5qw"
Sep 16 12:47:54.790055 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.790033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:47:54.797996 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.797977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2"
Sep 16 12:47:54.806682 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.806661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:47:54.815503 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.815483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jkzg"
Sep 16 12:47:54.825262 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.825241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmcvr"
Sep 16 12:47:54.834130 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.834109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p6wfm"
Sep 16 12:47:54.844012 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:54.843994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4h482"
Sep 16 12:47:55.071927 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:55.071889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:55.072085 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.072009    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:55.072125 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.072088    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:56.072067 +0000 UTC m=+4.022412536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:55.172877 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:55.172843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:55.173055 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:55.172890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:55.173055 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.173044    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:55.173172 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.173048    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:55.173172 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.173084    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:55.173172 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.173098    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:55.173172 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.173105    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:56.173089162 +0000 UTC m=+4.123434695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:55.173172 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.173163    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:47:56.173144743 +0000 UTC m=+4.123490281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:55.522791 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:55.522752    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:42:54 +0000 UTC" deadline="2028-04-03 04:22:59.821351062 +0000 UTC"
Sep 16 12:47:55.522791 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:55.522782    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13551h35m4.298572083s"
Sep 16 12:47:55.571138 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:55.571111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:55.571257 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:55.571236    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:47:55.611790 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:55.611766    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fb8233_0ed8_4d69_ba5f_0d998d7f2ca5.slice/crio-2343f8130176b5f8ea1a06b5d78c5e593d25ca9930d3ae010f268a43628c35f7 WatchSource:0}: Error finding container 2343f8130176b5f8ea1a06b5d78c5e593d25ca9930d3ae010f268a43628c35f7: Status 404 returned error can't find the container with id 2343f8130176b5f8ea1a06b5d78c5e593d25ca9930d3ae010f268a43628c35f7
Sep 16 12:47:55.621641 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:55.621612    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5759cfbb_d3e5_4129_b62e_00648ab43d8b.slice/crio-3f1cd58400869ca3bf6dc839d2a440acca7e70817c6b774c45042fb23cf6a5e8 WatchSource:0}: Error finding container 3f1cd58400869ca3bf6dc839d2a440acca7e70817c6b774c45042fb23cf6a5e8: Status 404 returned error can't find the container with id 3f1cd58400869ca3bf6dc839d2a440acca7e70817c6b774c45042fb23cf6a5e8
Sep 16 12:47:55.622485 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:55.622462    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588f3d87_3ab5_4b39_bb0b_2636e8411bb9.slice/crio-727fdee5f1ae7283507f0d71f834b0ac2fe79d59565faa867ac6433f2fcf5ea3 WatchSource:0}: Error finding container 727fdee5f1ae7283507f0d71f834b0ac2fe79d59565faa867ac6433f2fcf5ea3: Status 404 returned error can't find the container with id 727fdee5f1ae7283507f0d71f834b0ac2fe79d59565faa867ac6433f2fcf5ea3
Sep 16 12:47:55.623420 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:55.623402    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a8f2ab_0bc2_4dc7_b4bd_1acc18095840.slice/crio-b9a1823a85c864649e5a94b99d50f210d018327fa1fa04f0e4f593e8a00a9e76 WatchSource:0}: Error finding container b9a1823a85c864649e5a94b99d50f210d018327fa1fa04f0e4f593e8a00a9e76: Status 404 returned error can't find the container with id b9a1823a85c864649e5a94b99d50f210d018327fa1fa04f0e4f593e8a00a9e76
Sep 16 12:47:55.624887 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:55.624838    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce327f0_b212_4e77_a282_cd4e727bdc04.slice/crio-811aad5c901e6f69bac5a0db99c3a51bb57aa330201e6578e63d39a2bd034b4a WatchSource:0}: Error finding container 811aad5c901e6f69bac5a0db99c3a51bb57aa330201e6578e63d39a2bd034b4a: Status 404 returned error can't find the container with id 811aad5c901e6f69bac5a0db99c3a51bb57aa330201e6578e63d39a2bd034b4a
Sep 16 12:47:55.625833 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:47:55.625801    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9129bf1_95b9_47aa_83a7_2a76f3729b55.slice/crio-5d064adb0c19a7adef3ec237338287c159a363cad5bfeb21018f864c609896de WatchSource:0}: Error finding container 5d064adb0c19a7adef3ec237338287c159a363cad5bfeb21018f864c609896de: Status 404 returned error can't find the container with id 5d064adb0c19a7adef3ec237338287c159a363cad5bfeb21018f864c609896de
Sep 16 12:47:56.078491 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.078453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:56.078658 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.078619    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:56.078720 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.078702    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:58.078683825 +0000 UTC m=+6.029029380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:56.179834 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.179616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:56.179834 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.179744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:56.180087 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.179960    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:56.180087 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.179985    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:56.180087 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.180000    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:56.180087 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.180065    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:47:58.180044425 +0000 UTC m=+6.130389980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:56.182347 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.180479    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:56.182347 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.180537    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:58.180518767 +0000 UTC m=+6.130864314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:56.523293 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.523222    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:42:54 +0000 UTC" deadline="2028-03-31 18:33:47.783399258 +0000 UTC"
Sep 16 12:47:56.523293 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.523290    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13493h45m51.26011315s"
Sep 16 12:47:56.569555 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.568427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:56.569555 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.568616    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:47:56.569555 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.568995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:56.569555 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:56.569138    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:47:56.612399 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.612360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerStarted","Data":"46252866e9a3eedae91beede2acad97a97ee4554b1b2e5092bded16aa2d14d79"}
Sep 16 12:47:56.622906 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.622870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q46rc" event={"ID":"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e","Type":"ContainerStarted","Data":"8a7d4c77a327a001f47418b12c2e4b3125d42d8ba185ad72103ed81f50b1bff1"}
Sep 16 12:47:56.633718 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.633637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnvx4" event={"ID":"588f3d87-3ab5-4b39-bb0b-2636e8411bb9","Type":"ContainerStarted","Data":"727fdee5f1ae7283507f0d71f834b0ac2fe79d59565faa867ac6433f2fcf5ea3"}
Sep 16 12:47:56.641604 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.641293    2577 generic.go:356] "Generic (PLEG): container finished" podID="6aeab2e70341bb2c67f5334fcea3a0e2" containerID="736b85ca5b1d6a12a35b819b6ab053ec223e52d014625d1813f07d29181e176d" exitCode=0
Sep 16 12:47:56.641604 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.641383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal" event={"ID":"6aeab2e70341bb2c67f5334fcea3a0e2","Type":"ContainerDied","Data":"736b85ca5b1d6a12a35b819b6ab053ec223e52d014625d1813f07d29181e176d"}
Sep 16 12:47:56.655000 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.654843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2" event={"ID":"3cbe3b34-d894-4097-b400-96f05b0d077b","Type":"ContainerStarted","Data":"dc3d6dcdc07e10d045e3a0cdb4f64f21f959f6e2693163d95dee7afb488d795c"}
Sep 16 12:47:56.660895 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.660800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"5d064adb0c19a7adef3ec237338287c159a363cad5bfeb21018f864c609896de"}
Sep 16 12:47:56.666755 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.666728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jkzg" event={"ID":"5759cfbb-d3e5-4129-b62e-00648ab43d8b","Type":"ContainerStarted","Data":"3f1cd58400869ca3bf6dc839d2a440acca7e70817c6b774c45042fb23cf6a5e8"}
Sep 16 12:47:56.678916 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.678851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4h482" event={"ID":"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840","Type":"ContainerStarted","Data":"b9a1823a85c864649e5a94b99d50f210d018327fa1fa04f0e4f593e8a00a9e76"}
Sep 16 12:47:56.693652 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.693621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt5qw" event={"ID":"9ce327f0-b212-4e77-a282-cd4e727bdc04","Type":"ContainerStarted","Data":"811aad5c901e6f69bac5a0db99c3a51bb57aa330201e6578e63d39a2bd034b4a"}
Sep 16 12:47:56.705194 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.705142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p6wfm" event={"ID":"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5","Type":"ContainerStarted","Data":"2343f8130176b5f8ea1a06b5d78c5e593d25ca9930d3ae010f268a43628c35f7"}
Sep 16 12:47:56.718318 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.718282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal" event={"ID":"562cd238c1a07a23034785db4cb30206","Type":"ContainerStarted","Data":"dfdd87c0e113197adb921ba928070378df40a424d4bfea881148a9327ffe0ecb"}
Sep 16 12:47:56.735006 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:56.734952    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-99.ec2.internal" podStartSLOduration=2.7349171180000003 podStartE2EDuration="2.734917118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:56.73458493 +0000 UTC m=+4.684930480" watchObservedRunningTime="2026-09-16 12:47:56.734917118 +0000 UTC m=+4.685262675"
Sep 16 12:47:57.567737 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:57.567265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:57.567737 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:57.567399    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:47:57.737520 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:57.737435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal" event={"ID":"6aeab2e70341bb2c67f5334fcea3a0e2","Type":"ContainerStarted","Data":"96619484b7a388a8715ac00c58da2a894361df257d6dd277f6241f8b40bb355f"}
Sep 16 12:47:58.099782 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:58.099742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:58.099976 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.099954    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:58.100050 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.100031    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:02.100008838 +0000 UTC m=+10.050354374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:47:58.200847 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:58.200807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:58.201056 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:58.200919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:58.201153 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.201067    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:47:58.201153 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.201088    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:47:58.201153 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.201102    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:58.201308 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.201164    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:48:02.201142634 +0000 UTC m=+10.151488171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:47:58.201602 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.201581    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:58.201688 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.201640    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:02.201623317 +0000 UTC m=+10.151968857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:47:58.569721 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:58.569689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:47:58.570239 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.569823    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:47:58.570239 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:58.570228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:47:58.570357 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:58.570338    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:47:59.567527 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:47:59.567490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:47:59.567696 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:47:59.567609    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:00.569842 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:00.569238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:00.569842 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:00.569357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:00.569842 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:00.569677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:00.569842 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:00.569774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:01.567265 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:01.567035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:01.567425 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:01.567375    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:02.137229 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:02.137195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:02.137692 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.137301    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:02.137692 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.137347    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:10.137334665 +0000 UTC m=+18.087680196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:02.238001 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:02.237974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:02.238206 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:02.238024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:02.238206 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.238134    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:02.238206 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.238156    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:02.238206 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.238177    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:02.238206 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.238189    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:02.238206 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.238193    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:10.23817346 +0000 UTC m=+18.188519011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:02.238553 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.238231    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:48:10.238216087 +0000 UTC m=+18.188561623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:02.568546 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:02.568513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:02.568715 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.568634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:02.569127 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:02.568966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:02.569127 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:02.569082    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:03.567154 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:03.567117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:03.567529 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:03.567250    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:04.571030 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:04.571001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:04.571509 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:04.571037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:04.571509 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:04.571128    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:04.571509 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:04.571486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:05.567432 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:05.567388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:05.567579 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:05.567506    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:06.567157 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:06.567128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:06.567523 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:06.567240    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:06.567523 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:06.567327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:06.567523 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:06.567455    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:07.566942 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:07.566907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:07.567096 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:07.567014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:08.566783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:08.566748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:08.567251 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:08.566895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:08.567251 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:08.567244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:08.567362 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:08.567344    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:09.567400 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:09.567375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:09.567827 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:09.567493    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:10.200217 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:10.200182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:10.200396 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.200313    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:10.200396 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.200371    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:26.200350049 +0000 UTC m=+34.150695589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:10.300807 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:10.300772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:10.301013 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:10.300815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:10.301013 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.300939    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:10.301013 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.300960    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:10.301013 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.300980    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:10.301013 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.300991    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:10.301013 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.301006    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:26.300987878 +0000 UTC m=+34.251333429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:10.301259 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.301034    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:48:26.301019404 +0000 UTC m=+34.251364951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:10.567332 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:10.567246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:10.567332 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:10.567268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:10.567762 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.567392    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:10.567762 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:10.567526    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:11.567200 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:11.567169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:11.567396 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:11.567291    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:12.568126 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:12.568091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:12.568558 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:12.568281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:12.568687 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:12.568663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:12.568832 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:12.568812    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:13.567565 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:13.567533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:13.567740 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:13.567667    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:14.570753 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.570584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:14.571384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.570584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:14.571384 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:14.570837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:14.571384 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:14.570950    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:14.779502 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.779467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2" event={"ID":"3cbe3b34-d894-4097-b400-96f05b0d077b","Type":"ContainerStarted","Data":"005f2f775e9bbd93a0d5a931f411cbf1e3471d1f5c8e9eca4d485d3986efbd3c"}
Sep 16 12:48:14.783077 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.783052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"9cc20038da690852bb828dcbe8742baed5cf1ffb8a70dbe976918b813394e1f8"}
Sep 16 12:48:14.783268 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.783126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"4de3f53ddf51dc1eaed500b7f16d290c5787d80c1061b1d224d462495fea780b"}
Sep 16 12:48:14.783268 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.783176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"8ff5a7bc66414dbc043c95e1d3ae61471c67f1399479a2a387837936fe930ba6"}
Sep 16 12:48:14.783268 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.783220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"7c2bfcd43808036e2af69d77a46b7e84fa4071128708e018dca5c8a2883a3cf4"}
Sep 16 12:48:14.783268 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.783262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"82d2a5d85bbb84e43d3e92d993b1f341674a512b370641b2b948f4d0c72e2758"}
Sep 16 12:48:14.783438 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.783307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"cec5cdda622b35bb230f9135b7d493fbb95630788fc14aacb869d3a8130d946c"}
Sep 16 12:48:14.784784 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.784757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jkzg" event={"ID":"5759cfbb-d3e5-4129-b62e-00648ab43d8b","Type":"ContainerStarted","Data":"5a5000c1b02c1743cd682463ff68a741546e6af439e5122fc9e1969a525ca713"}
Sep 16 12:48:14.786968 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.786603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt5qw" event={"ID":"9ce327f0-b212-4e77-a282-cd4e727bdc04","Type":"ContainerStarted","Data":"61dc15b97d0f60017fb5edca2be927a39f7cc95708c8a563f4e8218796848dfc"}
Sep 16 12:48:14.788325 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.788300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p6wfm" event={"ID":"11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5","Type":"ContainerStarted","Data":"31fcd03249065c4203f51eca86dd19b4f840d4111c0b858007987bf2ccff6eaa"}
Sep 16 12:48:14.790043 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.790016    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff789be7-69c9-40bb-a8aa-42f34de68c8f" containerID="e61a902df2df93010682f540485bfe8ab3bcfeb11887215661e6ccf351cd96b0" exitCode=0
Sep 16 12:48:14.790154 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.790101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerDied","Data":"e61a902df2df93010682f540485bfe8ab3bcfeb11887215661e6ccf351cd96b0"}
Sep 16 12:48:14.792121 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.792094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q46rc" event={"ID":"3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e","Type":"ContainerStarted","Data":"ed8c3f6b50517d00dee4892634b83de4e7db65566bce23135b9a97409cc8cac7"}
Sep 16 12:48:14.793680 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.793659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnvx4" event={"ID":"588f3d87-3ab5-4b39-bb0b-2636e8411bb9","Type":"ContainerStarted","Data":"0acd7308d571f9666561d70147da5f70030ba54807a879e981f220e6b87102da"}
Sep 16 12:48:14.801639 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.801595    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8jkzg" podStartSLOduration=3.70706698 podStartE2EDuration="21.801579692s" totalImagesPullingTime="18.094512712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:14.801154601 +0000 UTC m=+22.751500155" watchObservedRunningTime="2026-09-16 12:48:14.801579692 +0000 UTC m=+22.751925264"
Sep 16 12:48:14.801735 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.801666    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-99.ec2.internal" podStartSLOduration=20.801660735 podStartE2EDuration="20.801660735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:57.762448152 +0000 UTC m=+5.712793712" watchObservedRunningTime="2026-09-16 12:48:14.801660735 +0000 UTC m=+22.752006303"
Sep 16 12:48:14.822380 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.822311    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jt5qw" podStartSLOduration=3.668145206 podStartE2EDuration="21.822300389s" totalImagesPullingTime="18.154155183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:14.821535924 +0000 UTC m=+22.771881477" watchObservedRunningTime="2026-09-16 12:48:14.822300389 +0000 UTC m=+22.772645941"
Sep 16 12:48:14.843102 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.843067    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xnvx4" podStartSLOduration=3.7509825980000002 podStartE2EDuration="21.843054283s" totalImagesPullingTime="18.092071685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:14.842943882 +0000 UTC m=+22.793289455" watchObservedRunningTime="2026-09-16 12:48:14.843054283 +0000 UTC m=+22.793399838"
Sep 16 12:48:14.873652 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.873608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q46rc" podStartSLOduration=3.785658862 podStartE2EDuration="21.873596831s" totalImagesPullingTime="18.087937969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:14.87334435 +0000 UTC m=+22.823689904" watchObservedRunningTime="2026-09-16 12:48:14.873596831 +0000 UTC m=+22.823942383"
Sep 16 12:48:14.924740 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:14.924696    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p6wfm" podStartSLOduration=3.813014096 podStartE2EDuration="21.924682656s" totalImagesPullingTime="18.11166856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:14.92463663 +0000 UTC m=+22.874982182" watchObservedRunningTime="2026-09-16 12:48:14.924682656 +0000 UTC m=+22.875028208"
Sep 16 12:48:15.412310 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.412137    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:48:15.529617 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.529518    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:48:15.41230365Z","UUID":"12da4c92-6848-4e38-9bb8-822763c00c65","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:48:15.531441 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.531407    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:48:15.531441 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.531444    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:48:15.567390 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.567298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:15.567514 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:15.567405    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:15.798413 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.798367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2" event={"ID":"3cbe3b34-d894-4097-b400-96f05b0d077b","Type":"ContainerStarted","Data":"60943cf4657e84e5d1c0d3b53dbe02e59812fc862030aecd527b70007f6ba815"}
Sep 16 12:48:15.800190 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.800157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4h482" event={"ID":"06a8f2ab-0bc2-4dc7-b4bd-1acc18095840","Type":"ContainerStarted","Data":"165b3518ef36066ea4313093440e6b3964519599d5a5ad5c3310aa9110857b67"}
Sep 16 12:48:15.817708 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:15.817629    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4h482" podStartSLOduration=4.725253598 podStartE2EDuration="22.817616825s" totalImagesPullingTime="18.092363227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:15.81709576 +0000 UTC m=+23.767441315" watchObservedRunningTime="2026-09-16 12:48:15.817616825 +0000 UTC m=+23.767962402"
Sep 16 12:48:16.567473 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:16.567443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:16.567613 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:16.567571    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:16.567672 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:16.567608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:16.567784 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:16.567752    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:16.806446 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:16.806404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"58dfada78c11f87b62263a722c4e827a883a28a7cd41a14fb84622ccb1e699b9"}
Sep 16 12:48:17.567024 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:17.566993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:17.567205 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:17.567097    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:17.811340 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:17.811270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2" event={"ID":"3cbe3b34-d894-4097-b400-96f05b0d077b","Type":"ContainerStarted","Data":"82c482968ae7fd4904f8faa8d90aafc6495717d55e1d40af90cd1491824f8dbb"}
Sep 16 12:48:17.833038 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:17.832934    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8p8p2" podStartSLOduration=4.11897235 podStartE2EDuration="24.832912094s" totalImagesPullingTime="20.713939744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:17.832740177 +0000 UTC m=+25.783085730" watchObservedRunningTime="2026-09-16 12:48:17.832912094 +0000 UTC m=+25.783257649"
Sep 16 12:48:18.164256 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:18.164178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:48:18.164871 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:18.164834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:48:18.567017 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:18.566958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:18.567188 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:18.567116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:18.567256 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:18.567216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:18.567335 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:18.567313    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:18.813726 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:18.813693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:48:18.814249 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:18.814232    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnvx4"
Sep 16 12:48:19.567874 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.567634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:19.568030 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:19.568005    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:19.817251 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.817225    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff789be7-69c9-40bb-a8aa-42f34de68c8f" containerID="3dd6134601bed232515e24c9c657a736af8877a0df69f2f64686f54e3b202fd1" exitCode=0
Sep 16 12:48:19.817664 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.817314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerDied","Data":"3dd6134601bed232515e24c9c657a736af8877a0df69f2f64686f54e3b202fd1"}
Sep 16 12:48:19.821273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.821251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" event={"ID":"d9129bf1-95b9-47aa-83a7-2a76f3729b55","Type":"ContainerStarted","Data":"39ca5f586ad52a04e588bdcd0602bf7f21103f45c5ed1b6449a89f157edc0324"}
Sep 16 12:48:19.821568 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.821552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:48:19.821672 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.821625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:48:19.821727 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.821679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:48:19.837356 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.837297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:48:19.837518 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.837504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:48:19.870806 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:19.870773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw" podStartSLOduration=8.727425837 podStartE2EDuration="26.870760388s" totalImagesPullingTime="18.143334551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:19.870588736 +0000 UTC m=+27.820934289" watchObservedRunningTime="2026-09-16 12:48:19.870760388 +0000 UTC m=+27.821105940"
Sep 16 12:48:20.566849 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:20.566814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:20.566849 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:20.566878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:20.567092 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:20.566996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:20.567161 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:20.567136    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:21.127898 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.127816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bj87h"]
Sep 16 12:48:21.128561 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.127941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:21.128561 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:21.128029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:21.130982 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.130958    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fr8w4"]
Sep 16 12:48:21.131079 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.131047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:21.131159 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:21.131122    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:21.131610 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.131588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mww2q"]
Sep 16 12:48:21.131706 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.131692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:21.131819 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:21.131798    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:21.827835 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.827806    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff789be7-69c9-40bb-a8aa-42f34de68c8f" containerID="c55a9895d09d1822221cf3868488065110ed9582f43441da7b65d56352703133" exitCode=0
Sep 16 12:48:21.828000 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:21.827890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerDied","Data":"c55a9895d09d1822221cf3868488065110ed9582f43441da7b65d56352703133"}
Sep 16 12:48:22.567782 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:22.567753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:22.568225 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:22.567841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:22.568225 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:22.567900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:22.568225 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:22.568015    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:22.568225 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:22.568053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:22.568225 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:22.568117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:23.834722 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:23.834682    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff789be7-69c9-40bb-a8aa-42f34de68c8f" containerID="3a30b5805302bc22460c30126ea7e276d6330fbb89b75a57481596750bdb7899" exitCode=0
Sep 16 12:48:23.835099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:23.834741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerDied","Data":"3a30b5805302bc22460c30126ea7e276d6330fbb89b75a57481596750bdb7899"}
Sep 16 12:48:24.567718 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:24.567692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:24.568027 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:24.567753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:24.568027 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:24.567757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:24.568027 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:24.567870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mww2q" podUID="8bb342f5-ba85-4bea-a1a5-fa75f526c896"
Sep 16 12:48:24.568027 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:24.567933    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bj87h" podUID="15cbea84-3f63-497b-b7db-7e88193b54d6"
Sep 16 12:48:24.568027 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:24.568004    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr8w4" podUID="4ffbc9a0-7600-4bcb-be04-63eab62a6cad"
Sep 16 12:48:26.223107 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.223071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:26.223642 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.223276    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:26.223642 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.223361    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret podName:15cbea84-3f63-497b-b7db-7e88193b54d6 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:58.223339811 +0000 UTC m=+66.173685361 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret") pod "global-pull-secret-syncer-bj87h" (UID: "15cbea84-3f63-497b-b7db-7e88193b54d6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:26.323549 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.323511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:26.323684 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.323586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:26.323684 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.323660    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:26.323802 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.323686    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:26.323802 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.323688    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:26.323802 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.323699    2577 projected.go:196] Error preparing data for projected volume kube-api-access-qlxdg for pod openshift-network-diagnostics/network-check-target-fr8w4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:26.323802 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.323754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:58.323736139 +0000 UTC m=+66.274081674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:26.323802 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.323769    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg podName:4ffbc9a0-7600-4bcb-be04-63eab62a6cad nodeName:}" failed. No retries permitted until 2026-09-16 12:48:58.323761388 +0000 UTC m=+66.274106919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlxdg" (UniqueName: "kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg") pod "network-check-target-fr8w4" (UID: "4ffbc9a0-7600-4bcb-be04-63eab62a6cad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:26.337458 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.337439    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-99.ec2.internal" event="NodeReady"
Sep 16 12:48:26.337580 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.337558    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:48:26.396012 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.395983    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l9s7f"]
Sep 16 12:48:26.398059 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.398032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.399212 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.399183    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9xxt"]
Sep 16 12:48:26.401773 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.401745    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:48:26.402009 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.401978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:48:26.403992 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.403966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dfdpm\""
Sep 16 12:48:26.404106 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.404015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.406558 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.406485    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:48:26.406662 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.406611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8lxmf\""
Sep 16 12:48:26.406725 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.406686    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:48:26.406926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.406909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:48:26.410825 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.410793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9s7f"]
Sep 16 12:48:26.413226 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.413195    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9xxt"]
Sep 16 12:48:26.524954 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.524850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.524954 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.524906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gkm\" (UniqueName: \"kubernetes.io/projected/78064f5c-911d-470b-9635-eafb7e7ee14c-kube-api-access-c4gkm\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.524954 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.524936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e825c3a2-5e2f-47be-8a80-0b0c74456208-cert\") pod \"ingress-canary-x9xxt\" (UID: \"e825c3a2-5e2f-47be-8a80-0b0c74456208\") " pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.525219 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.524985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltr7\" (UniqueName: \"kubernetes.io/projected/e825c3a2-5e2f-47be-8a80-0b0c74456208-kube-api-access-qltr7\") pod \"ingress-canary-x9xxt\" (UID: \"e825c3a2-5e2f-47be-8a80-0b0c74456208\") " pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.525219 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.525025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78064f5c-911d-470b-9635-eafb7e7ee14c-tmp-dir\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.525219 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.525104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/78064f5c-911d-470b-9635-eafb7e7ee14c-config-volume\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.567213 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.567178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:26.567562 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.567178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:26.568100 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.567178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:26.570578 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.570557    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:26.570999 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.570971    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:26.571121 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.571006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9nrdj\""
Sep 16 12:48:26.571190 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.571165    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:26.571247 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.571207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:26.571247 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.571165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m2lxw\""
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/78064f5c-911d-470b-9635-eafb7e7ee14c-config-volume\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gkm\" (UniqueName: \"kubernetes.io/projected/78064f5c-911d-470b-9635-eafb7e7ee14c-kube-api-access-c4gkm\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e825c3a2-5e2f-47be-8a80-0b0c74456208-cert\") pod \"ingress-canary-x9xxt\" (UID: \"e825c3a2-5e2f-47be-8a80-0b0c74456208\") " pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltr7\" (UniqueName: \"kubernetes.io/projected/e825c3a2-5e2f-47be-8a80-0b0c74456208-kube-api-access-qltr7\") pod \"ingress-canary-x9xxt\" (UID: \"e825c3a2-5e2f-47be-8a80-0b0c74456208\") " pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.626305    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78064f5c-911d-470b-9635-eafb7e7ee14c-tmp-dir\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:26.626380    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls podName:78064f5c-911d-470b-9635-eafb7e7ee14c nodeName:}" failed. No retries permitted until 2026-09-16 12:48:27.126356495 +0000 UTC m=+35.076702042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls") pod "dns-default-l9s7f" (UID: "78064f5c-911d-470b-9635-eafb7e7ee14c") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:26.626679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78064f5c-911d-470b-9635-eafb7e7ee14c-tmp-dir\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.627234 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.626764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/78064f5c-911d-470b-9635-eafb7e7ee14c-config-volume\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.631695 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.631660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e825c3a2-5e2f-47be-8a80-0b0c74456208-cert\") pod \"ingress-canary-x9xxt\" (UID: \"e825c3a2-5e2f-47be-8a80-0b0c74456208\") " pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.635592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.635573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gkm\" (UniqueName: \"kubernetes.io/projected/78064f5c-911d-470b-9635-eafb7e7ee14c-kube-api-access-c4gkm\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:26.635695 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.635678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltr7\" (UniqueName: \"kubernetes.io/projected/e825c3a2-5e2f-47be-8a80-0b0c74456208-kube-api-access-qltr7\") pod \"ingress-canary-x9xxt\" (UID: \"e825c3a2-5e2f-47be-8a80-0b0c74456208\") " pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:26.718804 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:26.718770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9xxt"
Sep 16 12:48:27.130573 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:27.130539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:27.130737 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:27.130675    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:27.130737 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:27.130730    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls podName:78064f5c-911d-470b-9635-eafb7e7ee14c nodeName:}" failed. No retries permitted until 2026-09-16 12:48:28.130716233 +0000 UTC m=+36.081061765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls") pod "dns-default-l9s7f" (UID: "78064f5c-911d-470b-9635-eafb7e7ee14c") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.056973 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:28.056905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9xxt"]
Sep 16 12:48:28.060725 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:28.060696    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode825c3a2_5e2f_47be_8a80_0b0c74456208.slice/crio-dcd495ee7dc9eb5d7f9c9dd559f971ec3b184301f979a3c72f848f9d090a1d79 WatchSource:0}: Error finding container dcd495ee7dc9eb5d7f9c9dd559f971ec3b184301f979a3c72f848f9d090a1d79: Status 404 returned error can't find the container with id dcd495ee7dc9eb5d7f9c9dd559f971ec3b184301f979a3c72f848f9d090a1d79
Sep 16 12:48:28.136560 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:28.136535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:28.136713 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:28.136695    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.136784 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:28.136770    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls podName:78064f5c-911d-470b-9635-eafb7e7ee14c nodeName:}" failed. No retries permitted until 2026-09-16 12:48:30.136749482 +0000 UTC m=+38.087095029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls") pod "dns-default-l9s7f" (UID: "78064f5c-911d-470b-9635-eafb7e7ee14c") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:28.850997 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:28.850962    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff789be7-69c9-40bb-a8aa-42f34de68c8f" containerID="9decd6ddafc9b7bba6ce19f21ed7d1cf75101dc3bdd529a4efd67328e5d17adb" exitCode=0
Sep 16 12:48:28.851155 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:28.851057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerDied","Data":"9decd6ddafc9b7bba6ce19f21ed7d1cf75101dc3bdd529a4efd67328e5d17adb"}
Sep 16 12:48:28.855695 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:28.855655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9xxt" event={"ID":"e825c3a2-5e2f-47be-8a80-0b0c74456208","Type":"ContainerStarted","Data":"dcd495ee7dc9eb5d7f9c9dd559f971ec3b184301f979a3c72f848f9d090a1d79"}
Sep 16 12:48:29.862327 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:29.862289    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff789be7-69c9-40bb-a8aa-42f34de68c8f" containerID="aa2ce6e31c43ca1dbe98bf455bcb821049dbdc88348b6eff200af98bca3b53a5" exitCode=0
Sep 16 12:48:29.862929 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:29.862337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerDied","Data":"aa2ce6e31c43ca1dbe98bf455bcb821049dbdc88348b6eff200af98bca3b53a5"}
Sep 16 12:48:30.151583 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:30.151493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:30.151738 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:30.151676    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:30.151798 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:30.151759    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls podName:78064f5c-911d-470b-9635-eafb7e7ee14c nodeName:}" failed. No retries permitted until 2026-09-16 12:48:34.151738263 +0000 UTC m=+42.102083817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls") pod "dns-default-l9s7f" (UID: "78064f5c-911d-470b-9635-eafb7e7ee14c") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:30.868068 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:30.867822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" event={"ID":"ff789be7-69c9-40bb-a8aa-42f34de68c8f","Type":"ContainerStarted","Data":"d786c7bc34248ae383f25e31bd72986971cd43aa1c92ca772103555901a9cbd1"}
Sep 16 12:48:30.869241 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:30.869210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9xxt" event={"ID":"e825c3a2-5e2f-47be-8a80-0b0c74456208","Type":"ContainerStarted","Data":"b94d73f55f72ead2ad26c6c068246c70f573008be0c202e8e7c24d84e4913c21"}
Sep 16 12:48:30.893606 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:30.893568    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xmcvr" podStartSLOduration=8.779230794 podStartE2EDuration="37.893552763s" totalImagesPullingTime="29.114321969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:30.893237473 +0000 UTC m=+38.843583025" watchObservedRunningTime="2026-09-16 12:48:30.893552763 +0000 UTC m=+38.843898316"
Sep 16 12:48:30.913570 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:30.913497    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9xxt" podStartSLOduration=2.586250653 podStartE2EDuration="4.91348142s" totalImagesPullingTime="2.327230767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:30.913276887 +0000 UTC m=+38.863622440" watchObservedRunningTime="2026-09-16 12:48:30.91348142 +0000 UTC m=+38.863826974"
Sep 16 12:48:34.180283 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:34.180249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:34.180712 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:34.180414    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:34.180712 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:34.180494    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls podName:78064f5c-911d-470b-9635-eafb7e7ee14c nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.18047033 +0000 UTC m=+50.130815876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls") pod "dns-default-l9s7f" (UID: "78064f5c-911d-470b-9635-eafb7e7ee14c") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:41.651759 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.651728    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mbvzm"]
Sep 16 12:48:41.664434 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.664409    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"]
Sep 16 12:48:41.664576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.664555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.675956 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.675932    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 12:48:41.676160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.676131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bgd7n\""
Sep 16 12:48:41.676304 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.676212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.676304 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.676004    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 12:48:41.676304 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.675976    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.684075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.684052    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-jgggh"]
Sep 16 12:48:41.684181 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.684167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.693322 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.693305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.693543 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.693529    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 12:48:41.695632 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.695618    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.701234 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.701212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 12:48:41.707955 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.701433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.707955 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.701446    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqknp\""
Sep 16 12:48:41.707955 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.702490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mbvzm"]
Sep 16 12:48:41.735334 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.735316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-jgggh"]
Sep 16 12:48:41.747845 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.747825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 12:48:41.754721 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.754706    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 12:48:41.756025 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.756011    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.756072 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.756036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cw7zk\""
Sep 16 12:48:41.756209 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.756072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 12:48:41.756376 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.756363    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.771743 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.771724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"]
Sep 16 12:48:41.778662 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.778647    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 12:48:41.799215 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.799194    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"]
Sep 16 12:48:41.806412 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.806396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:41.810025 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.810011    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 12:48:41.810102 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.810084    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 12:48:41.810537 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.810525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.811241 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.811226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.829036 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.829020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5mvc9\""
Sep 16 12:48:41.835116 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/088a59c6-e72f-409f-b775-369532584ed0-config\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.835211 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/088a59c6-e72f-409f-b775-369532584ed0-serving-cert\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.835211 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba4cd64-504f-4bae-9f8b-be8a24205fca-tmp\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.835211 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.835211 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba4cd64-504f-4bae-9f8b-be8a24205fca-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.835378 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4ba4cd64-504f-4bae-9f8b-be8a24205fca-snapshots\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.835378 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdl2c\" (UniqueName: \"kubernetes.io/projected/088a59c6-e72f-409f-b775-369532584ed0-kube-api-access-cdl2c\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.835378 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rt5t\" (UniqueName: \"kubernetes.io/projected/4ba4cd64-504f-4bae-9f8b-be8a24205fca-kube-api-access-7rt5t\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.835528 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f68d64d-28fc-411d-85ec-8fc331c3e756-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.835528 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/088a59c6-e72f-409f-b775-369532584ed0-trusted-ca\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.835627 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba4cd64-504f-4bae-9f8b-be8a24205fca-serving-cert\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.835627 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpfm\" (UniqueName: \"kubernetes.io/projected/4f68d64d-28fc-411d-85ec-8fc331c3e756-kube-api-access-6vpfm\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.835627 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.835589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba4cd64-504f-4bae-9f8b-be8a24205fca-service-ca-bundle\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.845765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.845735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"]
Sep 16 12:48:41.889635 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.889604    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"]
Sep 16 12:48:41.906929 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.906877    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74dfb48679-kcvt2"]
Sep 16 12:48:41.906929 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.906924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:41.912505 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.912481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 12:48:41.913576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.913542    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:41.913722 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.913700    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 12:48:41.913789 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.913775    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m6896\""
Sep 16 12:48:41.919431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.919404    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:41.919801 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.919784    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"]
Sep 16 12:48:41.919844 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.919819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74dfb48679-kcvt2"]
Sep 16 12:48:41.919919 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.919905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:41.922176 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.922158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:48:41.922246 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.922190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:48:41.922729 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.922713    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pwmxd\""
Sep 16 12:48:41.922813 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.922743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:48:41.935926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.935908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f68d64d-28fc-411d-85ec-8fc331c3e756-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.935997 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.935974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/088a59c6-e72f-409f-b775-369532584ed0-trusted-ca\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.935997 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.935991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba4cd64-504f-4bae-9f8b-be8a24205fca-serving-cert\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936084 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpfm\" (UniqueName: \"kubernetes.io/projected/4f68d64d-28fc-411d-85ec-8fc331c3e756-kube-api-access-6vpfm\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.936084 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba4cd64-504f-4bae-9f8b-be8a24205fca-service-ca-bundle\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936084 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/088a59c6-e72f-409f-b775-369532584ed0-config\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.936084 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/088a59c6-e72f-409f-b775-369532584ed0-serving-cert\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.936273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba4cd64-504f-4bae-9f8b-be8a24205fca-tmp\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.936273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-config\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:41.936273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba4cd64-504f-4bae-9f8b-be8a24205fca-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9s6\" (UniqueName: \"kubernetes.io/projected/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-kube-api-access-lz9s6\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:41.936273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4ba4cd64-504f-4bae-9f8b-be8a24205fca-snapshots\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936531 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdl2c\" (UniqueName: \"kubernetes.io/projected/088a59c6-e72f-409f-b775-369532584ed0-kube-api-access-cdl2c\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.936531 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rt5t\" (UniqueName: \"kubernetes.io/projected/4ba4cd64-504f-4bae-9f8b-be8a24205fca-kube-api-access-7rt5t\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936531 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-serving-cert\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:41.936671 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:41.936622    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:41.936763 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba4cd64-504f-4bae-9f8b-be8a24205fca-service-ca-bundle\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.936820 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.936742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f68d64d-28fc-411d-85ec-8fc331c3e756-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.936820 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:41.936753    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls podName:4f68d64d-28fc-411d-85ec-8fc331c3e756 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.43673185 +0000 UTC m=+50.387077408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfhlf" (UID: "4f68d64d-28fc-411d-85ec-8fc331c3e756") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:41.937138 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.937028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/088a59c6-e72f-409f-b775-369532584ed0-trusted-ca\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.937336 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.937314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/088a59c6-e72f-409f-b775-369532584ed0-config\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.937436 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.937377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba4cd64-504f-4bae-9f8b-be8a24205fca-tmp\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.937527 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.937500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4ba4cd64-504f-4bae-9f8b-be8a24205fca-snapshots\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.937689 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.937673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ba4cd64-504f-4bae-9f8b-be8a24205fca-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.938194 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.938174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:48:41.939944 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.939921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/088a59c6-e72f-409f-b775-369532584ed0-serving-cert\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.940066 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.940050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ba4cd64-504f-4bae-9f8b-be8a24205fca-serving-cert\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.948520 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.948497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdl2c\" (UniqueName: \"kubernetes.io/projected/088a59c6-e72f-409f-b775-369532584ed0-kube-api-access-cdl2c\") pod \"console-operator-696884fb9c-mbvzm\" (UID: \"088a59c6-e72f-409f-b775-369532584ed0\") " pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:41.951770 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.951747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpfm\" (UniqueName: \"kubernetes.io/projected/4f68d64d-28fc-411d-85ec-8fc331c3e756-kube-api-access-6vpfm\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:41.953204 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.953189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rt5t\" (UniqueName: \"kubernetes.io/projected/4ba4cd64-504f-4bae-9f8b-be8a24205fca-kube-api-access-7rt5t\") pod \"insights-operator-5844b856db-jgggh\" (UID: \"4ba4cd64-504f-4bae-9f8b-be8a24205fca\") " pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:41.973402 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:41.973381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:42.010096 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.010071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-jgggh"
Sep 16 12:48:42.037412 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-trusted-ca\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.037412 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-bound-sa-token\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.037412 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-image-registry-private-configuration\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.037631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-config\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.037631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-serving-cert\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.037631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bef9cefd-33b8-413a-b33f-e158ac86980e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.037631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.037631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0ba488e-ff53-4078-a1e6-a65634cc90b3-ca-trust-extracted\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.037631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7hd\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-kube-api-access-st7hd\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.038075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jdn\" (UniqueName: \"kubernetes.io/projected/bef9cefd-33b8-413a-b33f-e158ac86980e-kube-api-access-t4jdn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.038075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bef9cefd-33b8-413a-b33f-e158ac86980e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.038075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-installation-pull-secrets\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.038075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-config\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.038075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-certificates\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.038075 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.037898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9s6\" (UniqueName: \"kubernetes.io/projected/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-kube-api-access-lz9s6\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.042782 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.042763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-serving-cert\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.053802 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.053771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9s6\" (UniqueName: \"kubernetes.io/projected/98d5b95e-0e6b-47a1-a36f-8604c6db1e0a-kube-api-access-lz9s6\") pod \"service-ca-operator-cf7c776b7-lc5ln\" (UID: \"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.114161 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.114126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"
Sep 16 12:48:42.138687 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.138664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-installation-pull-secrets\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.138772 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.138706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-certificates\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.138772 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.138740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-trusted-ca\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.138915 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.138896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-bound-sa-token\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.138974 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.138940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-image-registry-private-configuration\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.139025 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.138984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bef9cefd-33b8-413a-b33f-e158ac86980e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.139025 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.139112 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0ba488e-ff53-4078-a1e6-a65634cc90b3-ca-trust-extracted\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.139112 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7hd\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-kube-api-access-st7hd\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.139112 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jdn\" (UniqueName: \"kubernetes.io/projected/bef9cefd-33b8-413a-b33f-e158ac86980e-kube-api-access-t4jdn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.139245 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.139116    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:42.139245 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bef9cefd-33b8-413a-b33f-e158ac86980e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.139245 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.139135    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dfb48679-kcvt2: secret "image-registry-tls" not found
Sep 16 12:48:42.139245 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.139210    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls podName:f0ba488e-ff53-4078-a1e6-a65634cc90b3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.639188238 +0000 UTC m=+50.589533790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls") pod "image-registry-74dfb48679-kcvt2" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3") : secret "image-registry-tls" not found
Sep 16 12:48:42.139812 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-certificates\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.139812 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.139561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0ba488e-ff53-4078-a1e6-a65634cc90b3-ca-trust-extracted\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.140070 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.140045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-trusted-ca\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.140181 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.140154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bef9cefd-33b8-413a-b33f-e158ac86980e-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.141735 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.141712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bef9cefd-33b8-413a-b33f-e158ac86980e-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.141919 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.141885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-image-registry-private-configuration\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.142070 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.142055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-installation-pull-secrets\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.168287 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.168212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-bound-sa-token\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.169505 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.169485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jdn\" (UniqueName: \"kubernetes.io/projected/bef9cefd-33b8-413a-b33f-e158ac86980e-kube-api-access-t4jdn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gj6m9\" (UID: \"bef9cefd-33b8-413a-b33f-e158ac86980e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.169895 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.169850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7hd\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-kube-api-access-st7hd\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.218557 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.218533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"
Sep 16 12:48:42.239671 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.239649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:42.239823 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.239799    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:42.239938 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.239896    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls podName:78064f5c-911d-470b-9635-eafb7e7ee14c nodeName:}" failed. No retries permitted until 2026-09-16 12:48:58.239873347 +0000 UTC m=+66.190218892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls") pod "dns-default-l9s7f" (UID: "78064f5c-911d-470b-9635-eafb7e7ee14c") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:42.259272 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.259247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mbvzm"]
Sep 16 12:48:42.263619 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:42.263593    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088a59c6_e72f_409f_b775_369532584ed0.slice/crio-1ebb80726546f4e4ddfc94e4ab6b9208415044a0dc54143b3c42ddda690926e5 WatchSource:0}: Error finding container 1ebb80726546f4e4ddfc94e4ab6b9208415044a0dc54143b3c42ddda690926e5: Status 404 returned error can't find the container with id 1ebb80726546f4e4ddfc94e4ab6b9208415044a0dc54143b3c42ddda690926e5
Sep 16 12:48:42.307669 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.307647    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-jgggh"]
Sep 16 12:48:42.310391 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:42.310366    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4cd64_504f_4bae_9f8b_be8a24205fca.slice/crio-89d88d2b24511dc63453f59eeec82c206b0cb182f43d570c63b1513fb07a8cca WatchSource:0}: Error finding container 89d88d2b24511dc63453f59eeec82c206b0cb182f43d570c63b1513fb07a8cca: Status 404 returned error can't find the container with id 89d88d2b24511dc63453f59eeec82c206b0cb182f43d570c63b1513fb07a8cca
Sep 16 12:48:42.400332 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.400307    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln"]
Sep 16 12:48:42.403179 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:42.403156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d5b95e_0e6b_47a1_a36f_8604c6db1e0a.slice/crio-2b82325e49b57cf13ec4b935873c0cd3d842d3e511cb789e888704d278e43b81 WatchSource:0}: Error finding container 2b82325e49b57cf13ec4b935873c0cd3d842d3e511cb789e888704d278e43b81: Status 404 returned error can't find the container with id 2b82325e49b57cf13ec4b935873c0cd3d842d3e511cb789e888704d278e43b81
Sep 16 12:48:42.442015 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.441989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:42.442111 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.442096    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:42.442155 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.442149    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls podName:4f68d64d-28fc-411d-85ec-8fc331c3e756 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.442135966 +0000 UTC m=+51.392481497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfhlf" (UID: "4f68d64d-28fc-411d-85ec-8fc331c3e756") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:42.515707 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.515678    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9"]
Sep 16 12:48:42.518221 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:42.518199    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef9cefd_33b8_413a_b33f_e158ac86980e.slice/crio-e8963f728131d78c6fd08f8d202e098b861243cf78a189a53cc07b4088cf5f3f WatchSource:0}: Error finding container e8963f728131d78c6fd08f8d202e098b861243cf78a189a53cc07b4088cf5f3f: Status 404 returned error can't find the container with id e8963f728131d78c6fd08f8d202e098b861243cf78a189a53cc07b4088cf5f3f
Sep 16 12:48:42.643648 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.643617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:42.643831 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.643807    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:42.643924 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.643835    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dfb48679-kcvt2: secret "image-registry-tls" not found
Sep 16 12:48:42.643989 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:42.643936    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls podName:f0ba488e-ff53-4078-a1e6-a65634cc90b3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:43.643914485 +0000 UTC m=+51.594260035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls") pod "image-registry-74dfb48679-kcvt2" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3") : secret "image-registry-tls" not found
Sep 16 12:48:42.901203 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.901103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln" event={"ID":"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a","Type":"ContainerStarted","Data":"2b82325e49b57cf13ec4b935873c0cd3d842d3e511cb789e888704d278e43b81"}
Sep 16 12:48:42.902730 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.902696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9" event={"ID":"bef9cefd-33b8-413a-b33f-e158ac86980e","Type":"ContainerStarted","Data":"e8963f728131d78c6fd08f8d202e098b861243cf78a189a53cc07b4088cf5f3f"}
Sep 16 12:48:42.904472 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.904441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jgggh" event={"ID":"4ba4cd64-504f-4bae-9f8b-be8a24205fca","Type":"ContainerStarted","Data":"89d88d2b24511dc63453f59eeec82c206b0cb182f43d570c63b1513fb07a8cca"}
Sep 16 12:48:42.906693 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:42.906669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" event={"ID":"088a59c6-e72f-409f-b775-369532584ed0","Type":"ContainerStarted","Data":"1ebb80726546f4e4ddfc94e4ab6b9208415044a0dc54143b3c42ddda690926e5"}
Sep 16 12:48:43.451772 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:43.451733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:43.451998 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:43.451881    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:43.451998 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:43.451952    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls podName:4f68d64d-28fc-411d-85ec-8fc331c3e756 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:45.451930823 +0000 UTC m=+53.402276371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfhlf" (UID: "4f68d64d-28fc-411d-85ec-8fc331c3e756") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:43.653335 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:43.653297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:43.653510 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:43.653500    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:43.653574 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:43.653516    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dfb48679-kcvt2: secret "image-registry-tls" not found
Sep 16 12:48:43.653636 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:43.653580    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls podName:f0ba488e-ff53-4078-a1e6-a65634cc90b3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:45.653560762 +0000 UTC m=+53.603906306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls") pod "image-registry-74dfb48679-kcvt2" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3") : secret "image-registry-tls" not found
Sep 16 12:48:45.469272 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.469221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:45.469844 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:45.469368    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:45.469844 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:45.469457    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls podName:4f68d64d-28fc-411d-85ec-8fc331c3e756 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.46943406 +0000 UTC m=+57.419779605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfhlf" (UID: "4f68d64d-28fc-411d-85ec-8fc331c3e756") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:45.670358 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.670322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:45.670514 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:45.670449    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:45.670514 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:45.670459    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dfb48679-kcvt2: secret "image-registry-tls" not found
Sep 16 12:48:45.670514 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:45.670507    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls podName:f0ba488e-ff53-4078-a1e6-a65634cc90b3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.670494177 +0000 UTC m=+57.620839722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls") pod "image-registry-74dfb48679-kcvt2" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3") : secret "image-registry-tls" not found
Sep 16 12:48:45.917836 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.917770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9" event={"ID":"bef9cefd-33b8-413a-b33f-e158ac86980e","Type":"ContainerStarted","Data":"d8cbfd2cb7ae4be0fac12f0927ba92dcc43181ebcc5f38f51d37e5c91a679210"}
Sep 16 12:48:45.919535 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.919513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jgggh" event={"ID":"4ba4cd64-504f-4bae-9f8b-be8a24205fca","Type":"ContainerStarted","Data":"a1e90c0a0ad50e8af19f7354495a217852c547e235d726f19c6f0a3c8fcc7bf3"}
Sep 16 12:48:45.921379 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.921350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/0.log"
Sep 16 12:48:45.921486 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.921393    2577 generic.go:356] "Generic (PLEG): container finished" podID="088a59c6-e72f-409f-b775-369532584ed0" containerID="23a2665796a72473d47fde60a0f7f3ff60a387684783c488b6464db3253862c3" exitCode=255
Sep 16 12:48:45.921544 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.921487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" event={"ID":"088a59c6-e72f-409f-b775-369532584ed0","Type":"ContainerDied","Data":"23a2665796a72473d47fde60a0f7f3ff60a387684783c488b6464db3253862c3"}
Sep 16 12:48:45.921784 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.921742    2577 scope.go:117] "RemoveContainer" containerID="23a2665796a72473d47fde60a0f7f3ff60a387684783c488b6464db3253862c3"
Sep 16 12:48:45.924359 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.924337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln" event={"ID":"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a","Type":"ContainerStarted","Data":"ef992bd10df1bd005349f2fdaa3c73b9af78de80a0f5b38328a6894f765ad326"}
Sep 16 12:48:45.938148 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.937886    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9" podStartSLOduration=1.686323705 podStartE2EDuration="4.937846597s" totalImagesPullingTime="3.251522892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:45.93754438 +0000 UTC m=+53.887889935" watchObservedRunningTime="2026-09-16 12:48:45.937846597 +0000 UTC m=+53.888192146"
Sep 16 12:48:45.990197 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:45.990153    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-jgggh" podStartSLOduration=1.529253136 podStartE2EDuration="4.990133776s" totalImagesPullingTime="3.46088064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:45.989495606 +0000 UTC m=+53.939841177" watchObservedRunningTime="2026-09-16 12:48:45.990133776 +0000 UTC m=+53.940479332"
Sep 16 12:48:46.025178 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.025049    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln" podStartSLOduration=1.6591221649999999 podStartE2EDuration="5.025027892s" totalImagesPullingTime="3.365905727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:46.023373708 +0000 UTC m=+53.973719262" watchObservedRunningTime="2026-09-16 12:48:46.025027892 +0000 UTC m=+53.975373447"
Sep 16 12:48:46.930898 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.930843    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:48:46.931522 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.931498    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/0.log"
Sep 16 12:48:46.931639 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.931536    2577 generic.go:356] "Generic (PLEG): container finished" podID="088a59c6-e72f-409f-b775-369532584ed0" containerID="2d337e2734f2c0afe378cf513b58205634fa4be3066bc3129daee0a9494a13af" exitCode=255
Sep 16 12:48:46.933708 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.932362    2577 scope.go:117] "RemoveContainer" containerID="2d337e2734f2c0afe378cf513b58205634fa4be3066bc3129daee0a9494a13af"
Sep 16 12:48:46.933708 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:46.932553    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mbvzm_openshift-console-operator(088a59c6-e72f-409f-b775-369532584ed0)\"" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" podUID="088a59c6-e72f-409f-b775-369532584ed0"
Sep 16 12:48:46.933708 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.932591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" event={"ID":"088a59c6-e72f-409f-b775-369532584ed0","Type":"ContainerDied","Data":"2d337e2734f2c0afe378cf513b58205634fa4be3066bc3129daee0a9494a13af"}
Sep 16 12:48:46.933708 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:46.932631    2577 scope.go:117] "RemoveContainer" containerID="23a2665796a72473d47fde60a0f7f3ff60a387684783c488b6464db3253862c3"
Sep 16 12:48:47.017791 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.017761    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tjwqw"]
Sep 16 12:48:47.020225 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.020207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.023960 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.023928    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-8zmc7\" is forbidden: User \"system:node:ip-10-0-143-99.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zmc7\"" type="*v1.Secret"
Sep 16 12:48:47.025309 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.025129    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-143-99.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Sep 16 12:48:47.025309 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.025141    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-tjwqw\" is forbidden: User \"system:node:ip-10-0-143-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-99.ec2.internal' and this object" podUID="381d5120-3a5b-43b1-af75-56abb87e27b5" pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.025309 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.025231    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-143-99.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-143-99.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Sep 16 12:48:47.045978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.045953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tjwqw"]
Sep 16 12:48:47.181876 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.181780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.182004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.181879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/381d5120-3a5b-43b1-af75-56abb87e27b5-crio-socket\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.182004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.181916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6cfb\" (UniqueName: \"kubernetes.io/projected/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-api-access-f6cfb\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.182004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.181949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.182004 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.182001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/381d5120-3a5b-43b1-af75-56abb87e27b5-data-volume\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283102 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/381d5120-3a5b-43b1-af75-56abb87e27b5-data-volume\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283275 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283275 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/381d5120-3a5b-43b1-af75-56abb87e27b5-crio-socket\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283275 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6cfb\" (UniqueName: \"kubernetes.io/projected/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-api-access-f6cfb\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283394 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283394 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/381d5120-3a5b-43b1-af75-56abb87e27b5-crio-socket\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.283584 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.283553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/381d5120-3a5b-43b1-af75-56abb87e27b5-data-volume\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.304401 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.304364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6cfb\" (UniqueName: \"kubernetes.io/projected/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-api-access-f6cfb\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:47.936024 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.935993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:48:47.936408 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.936355    2577 scope.go:117] "RemoveContainer" containerID="2d337e2734f2c0afe378cf513b58205634fa4be3066bc3129daee0a9494a13af"
Sep 16 12:48:47.936540 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.936520    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mbvzm_openshift-console-operator(088a59c6-e72f-409f-b775-369532584ed0)\"" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" podUID="088a59c6-e72f-409f-b775-369532584ed0"
Sep 16 12:48:47.958506 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:47.958481    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:48:47.963626 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.963610    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:47.963722 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:47.963677    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:48.463660663 +0000 UTC m=+56.414006194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:48.235076 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:48.235046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zmc7\""
Sep 16 12:48:48.283734 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:48.283710    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Sep 16 12:48:48.283880 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:48.283796    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:48.783760283 +0000 UTC m=+56.734105818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : failed to sync secret cache: timed out waiting for the condition
Sep 16 12:48:48.442447 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:48.442423    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:48:48.494262 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:48.494200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:48.494380 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:48.494323    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:48.494416 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:48.494382    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.494367394 +0000 UTC m=+57.444712942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:48.796342 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:48.796268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:48.796480 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:48.796414    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:48.796480 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:48.796474    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.79645972 +0000 UTC m=+57.746805251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:48.929139 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:48.929115    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p6wfm_11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5/dns-node-resolver/0.log"
Sep 16 12:48:49.501888 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.501839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:49.502263 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.501903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:49.502263 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.501974    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:49.502263 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.501994    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:49.502263 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.502028    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:51.502013311 +0000 UTC m=+59.452358842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:49.502263 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.502064    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls podName:4f68d64d-28fc-411d-85ec-8fc331c3e756 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:57.502050404 +0000 UTC m=+65.452395939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfhlf" (UID: "4f68d64d-28fc-411d-85ec-8fc331c3e756") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:49.702943 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.702911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:49.703095 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.703068    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:48:49.703095 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.703088    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dfb48679-kcvt2: secret "image-registry-tls" not found
Sep 16 12:48:49.703168 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.703153    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls podName:f0ba488e-ff53-4078-a1e6-a65634cc90b3 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:57.703137871 +0000 UTC m=+65.653483405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls") pod "image-registry-74dfb48679-kcvt2" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3") : secret "image-registry-tls" not found
Sep 16 12:48:49.770454 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.770382    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-drzfx"]
Sep 16 12:48:49.772505 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.772487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:49.776536 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.776517    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 12:48:49.776636 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.776580    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 12:48:49.777678 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.777662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w7n5c\""
Sep 16 12:48:49.777762 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.777695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 12:48:49.777762 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.777723    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 12:48:49.804058 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.804033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:49.804184 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.804167    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:49.804242 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:49.804227    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:51.804213241 +0000 UTC m=+59.754558773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:49.805151 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.805132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-drzfx"]
Sep 16 12:48:49.904671 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.904645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b33a67ae-a361-4763-8593-7139e8835a0c-signing-cabundle\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:49.904836 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.904680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b33a67ae-a361-4763-8593-7139e8835a0c-signing-key\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:49.904836 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.904706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2rz\" (UniqueName: \"kubernetes.io/projected/b33a67ae-a361-4763-8593-7139e8835a0c-kube-api-access-wn2rz\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:49.931670 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:49.931648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jkzg_5759cfbb-d3e5-4129-b62e-00648ab43d8b/node-ca/0.log"
Sep 16 12:48:50.006090 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.006064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b33a67ae-a361-4763-8593-7139e8835a0c-signing-cabundle\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.006212 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.006094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b33a67ae-a361-4763-8593-7139e8835a0c-signing-key\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.006212 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.006119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2rz\" (UniqueName: \"kubernetes.io/projected/b33a67ae-a361-4763-8593-7139e8835a0c-kube-api-access-wn2rz\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.006663 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.006644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b33a67ae-a361-4763-8593-7139e8835a0c-signing-cabundle\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.008537 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.008518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b33a67ae-a361-4763-8593-7139e8835a0c-signing-key\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.018278 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.018257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2rz\" (UniqueName: \"kubernetes.io/projected/b33a67ae-a361-4763-8593-7139e8835a0c-kube-api-access-wn2rz\") pod \"service-ca-568869fdc5-drzfx\" (UID: \"b33a67ae-a361-4763-8593-7139e8835a0c\") " pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.081257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.081198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-drzfx"
Sep 16 12:48:50.368283 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.368195    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-drzfx"]
Sep 16 12:48:50.372029 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:50.372002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33a67ae_a361_4763_8593_7139e8835a0c.slice/crio-8890706fc4e63fd9d5724f12a0a0c64e719bb789cce875776b68c650207a0755 WatchSource:0}: Error finding container 8890706fc4e63fd9d5724f12a0a0c64e719bb789cce875776b68c650207a0755: Status 404 returned error can't find the container with id 8890706fc4e63fd9d5724f12a0a0c64e719bb789cce875776b68c650207a0755
Sep 16 12:48:50.940160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.940132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9xxt_e825c3a2-5e2f-47be-8a80-0b0c74456208/serve-healthcheck-canary/0.log"
Sep 16 12:48:50.945262 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.945233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-drzfx" event={"ID":"b33a67ae-a361-4763-8593-7139e8835a0c","Type":"ContainerStarted","Data":"bdaf01d5c23c191d34d1e81c45e6f9222aee1c057da2e9224f370b451e226b0e"}
Sep 16 12:48:50.945414 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.945286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-drzfx" event={"ID":"b33a67ae-a361-4763-8593-7139e8835a0c","Type":"ContainerStarted","Data":"8890706fc4e63fd9d5724f12a0a0c64e719bb789cce875776b68c650207a0755"}
Sep 16 12:48:50.974968 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:50.974925    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-drzfx" podStartSLOduration=1.974909506 podStartE2EDuration="1.974909506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.974754626 +0000 UTC m=+58.925100179" watchObservedRunningTime="2026-09-16 12:48:50.974909506 +0000 UTC m=+58.925255071"
Sep 16 12:48:51.517127 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:51.517096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:51.517272 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:51.517181    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:51.517272 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:51.517234    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:55.517218735 +0000 UTC m=+63.467564270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:51.818991 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:51.818914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:51.819129 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:51.819065    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:51.819172 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:51.819147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:55.81913346 +0000 UTC m=+63.769479003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:51.849639 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:51.849608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8psw"
Sep 16 12:48:51.973518 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:51.973485    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:51.974003 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:51.973577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:48:51.974003 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:51.973953    2577 scope.go:117] "RemoveContainer" containerID="2d337e2734f2c0afe378cf513b58205634fa4be3066bc3129daee0a9494a13af"
Sep 16 12:48:51.974139 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:51.974118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mbvzm_openshift-console-operator(088a59c6-e72f-409f-b775-369532584ed0)\"" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" podUID="088a59c6-e72f-409f-b775-369532584ed0"
Sep 16 12:48:55.549913 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:55.549877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:55.550263 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:55.549992    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:48:55.550263 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:55.550055    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:03.550039188 +0000 UTC m=+71.500384718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : configmap "kube-rbac-proxy" not found
Sep 16 12:48:55.852358 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:55.852262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:48:55.852497 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:55.852423    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:55.852497 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:55.852491    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls podName:381d5120-3a5b-43b1-af75-56abb87e27b5 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:03.852474373 +0000 UTC m=+71.802819908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tjwqw" (UID: "381d5120-3a5b-43b1-af75-56abb87e27b5") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:48:57.565370 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:57.565338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:48:57.565765 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:57.565440    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:57.565765 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:57.565490    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls podName:4f68d64d-28fc-411d-85ec-8fc331c3e756 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:13.565476998 +0000 UTC m=+81.515822529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfhlf" (UID: "4f68d64d-28fc-411d-85ec-8fc331c3e756") : secret "cluster-monitoring-operator-tls" not found
Sep 16 12:48:57.767354 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:57.767321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:57.769734 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:57.769716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"image-registry-74dfb48679-kcvt2\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") " pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:57.829692 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:57.829635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pwmxd\""
Sep 16 12:48:57.837769 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:57.837754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:58.139093 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.139020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74dfb48679-kcvt2"]
Sep 16 12:48:58.142478 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:58.142440    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ba488e_ff53_4078_a1e6_a65634cc90b3.slice/crio-51c056ef9a41d04b0af9a67cc67bab81effd40bd81cd10406d6c549505b381b5 WatchSource:0}: Error finding container 51c056ef9a41d04b0af9a67cc67bab81effd40bd81cd10406d6c549505b381b5: Status 404 returned error can't find the container with id 51c056ef9a41d04b0af9a67cc67bab81effd40bd81cd10406d6c549505b381b5
Sep 16 12:48:58.271522 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.271498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:58.271625 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.271556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:58.273926 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.273902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78064f5c-911d-470b-9635-eafb7e7ee14c-metrics-tls\") pod \"dns-default-l9s7f\" (UID: \"78064f5c-911d-470b-9635-eafb7e7ee14c\") " pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:58.274130 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.274114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:58.284299 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.284280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15cbea84-3f63-497b-b7db-7e88193b54d6-original-pull-secret\") pod \"global-pull-secret-syncer-bj87h\" (UID: \"15cbea84-3f63-497b-b7db-7e88193b54d6\") " pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:58.372342 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.372317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:58.372456 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.372358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:48:58.375428 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.375412    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:58.375669 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.375651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:58.383502 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:58.383478    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:48:58.383590 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:48:58.383536    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs podName:8bb342f5-ba85-4bea-a1a5-fa75f526c896 nodeName:}" failed. No retries permitted until 2026-09-16 12:50:02.383516351 +0000 UTC m=+130.333861882 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs") pod "network-metrics-daemon-mww2q" (UID: "8bb342f5-ba85-4bea-a1a5-fa75f526c896") : secret "metrics-daemon-secret" not found
Sep 16 12:48:58.385744 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.385729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bj87h"
Sep 16 12:48:58.387271 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.387257    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:58.396515 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.396466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxdg\" (UniqueName: \"kubernetes.io/projected/4ffbc9a0-7600-4bcb-be04-63eab62a6cad-kube-api-access-qlxdg\") pod \"network-check-target-fr8w4\" (UID: \"4ffbc9a0-7600-4bcb-be04-63eab62a6cad\") " pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:58.515050 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.515022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dfdpm\""
Sep 16 12:48:58.522394 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.522371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:48:58.674204 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.674133    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bj87h"]
Sep 16 12:48:58.677204 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:58.677175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cbea84_3f63_497b_b7db_7e88193b54d6.slice/crio-4821213756f786c283859c00404fff0d60be0e5375e300c097d3aa0d1fe950d2 WatchSource:0}: Error finding container 4821213756f786c283859c00404fff0d60be0e5375e300c097d3aa0d1fe950d2: Status 404 returned error can't find the container with id 4821213756f786c283859c00404fff0d60be0e5375e300c097d3aa0d1fe950d2
Sep 16 12:48:58.681648 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.681627    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m2lxw\""
Sep 16 12:48:58.688767 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.688748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:48:58.824565 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.824538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9s7f"]
Sep 16 12:48:58.827810 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:58.827786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78064f5c_911d_470b_9635_eafb7e7ee14c.slice/crio-e7e8e84cb6da9380de7d467c8988a01a70298da3c7d446e6bdac5726954ab58d WatchSource:0}: Error finding container e7e8e84cb6da9380de7d467c8988a01a70298da3c7d446e6bdac5726954ab58d: Status 404 returned error can't find the container with id e7e8e84cb6da9380de7d467c8988a01a70298da3c7d446e6bdac5726954ab58d
Sep 16 12:48:58.976028 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.975998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9s7f" event={"ID":"78064f5c-911d-470b-9635-eafb7e7ee14c","Type":"ContainerStarted","Data":"e7e8e84cb6da9380de7d467c8988a01a70298da3c7d446e6bdac5726954ab58d"}
Sep 16 12:48:58.977260 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.977237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bj87h" event={"ID":"15cbea84-3f63-497b-b7db-7e88193b54d6","Type":"ContainerStarted","Data":"4821213756f786c283859c00404fff0d60be0e5375e300c097d3aa0d1fe950d2"}
Sep 16 12:48:58.978631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.978608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" event={"ID":"f0ba488e-ff53-4078-a1e6-a65634cc90b3","Type":"ContainerStarted","Data":"a8ad68bb17ddf189d2270148857b02d7f64c4118e97a1ef81880ac349538614c"}
Sep 16 12:48:58.978714 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.978646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" event={"ID":"f0ba488e-ff53-4078-a1e6-a65634cc90b3","Type":"ContainerStarted","Data":"51c056ef9a41d04b0af9a67cc67bab81effd40bd81cd10406d6c549505b381b5"}
Sep 16 12:48:58.978811 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.978795    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:48:58.990089 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:58.990070    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fr8w4"]
Sep 16 12:48:58.993069 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:48:58.993050    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffbc9a0_7600_4bcb_be04_63eab62a6cad.slice/crio-d65cb0bed3712e3f24d5861c00bbce1facb4bd891d5eabb56d2ea093c3126aec WatchSource:0}: Error finding container d65cb0bed3712e3f24d5861c00bbce1facb4bd891d5eabb56d2ea093c3126aec: Status 404 returned error can't find the container with id d65cb0bed3712e3f24d5861c00bbce1facb4bd891d5eabb56d2ea093c3126aec
Sep 16 12:48:59.018368 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:59.018332    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" podStartSLOduration=18.018319763 podStartE2EDuration="18.018319763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:59.016997177 +0000 UTC m=+66.967342742" watchObservedRunningTime="2026-09-16 12:48:59.018319763 +0000 UTC m=+66.968665316"
Sep 16 12:48:59.987050 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:48:59.987011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fr8w4" event={"ID":"4ffbc9a0-7600-4bcb-be04-63eab62a6cad","Type":"ContainerStarted","Data":"d65cb0bed3712e3f24d5861c00bbce1facb4bd891d5eabb56d2ea093c3126aec"}
Sep 16 12:49:00.992519 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:00.992491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9s7f" event={"ID":"78064f5c-911d-470b-9635-eafb7e7ee14c","Type":"ContainerStarted","Data":"5bff2611f39f8bba974326a2b0f7d346f71c177875fa334bd500f515ec571e98"}
Sep 16 12:49:02.001714 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:02.001678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9s7f" event={"ID":"78064f5c-911d-470b-9635-eafb7e7ee14c","Type":"ContainerStarted","Data":"23812cccbc6a179cb2e9ca6f9c672c94dc24524747e57d29d7531ccbd64dfdb8"}
Sep 16 12:49:02.002541 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:02.002519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:49:02.027153 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:02.027109    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l9s7f" podStartSLOduration=34.092214269 podStartE2EDuration="36.027095115s" totalImagesPullingTime="1.934880846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:02.026766314 +0000 UTC m=+69.977111867" watchObservedRunningTime="2026-09-16 12:49:02.027095115 +0000 UTC m=+69.977440667"
Sep 16 12:49:03.618747 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:03.618728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:49:03.619363 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:03.619338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/381d5120-3a5b-43b1-af75-56abb87e27b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:49:03.921693 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:03.921621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:49:03.924005 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:03.923980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/381d5120-3a5b-43b1-af75-56abb87e27b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tjwqw\" (UID: \"381d5120-3a5b-43b1-af75-56abb87e27b5\") " pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:49:04.009840 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.009806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bj87h" event={"ID":"15cbea84-3f63-497b-b7db-7e88193b54d6","Type":"ContainerStarted","Data":"261640e229c526603a146d5c03ce367fdab708971e48ccf1136e50692c74ba14"}
Sep 16 12:49:04.015184 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.015148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fr8w4" event={"ID":"4ffbc9a0-7600-4bcb-be04-63eab62a6cad","Type":"ContainerStarted","Data":"991e0c59e999ec2bc6957f6defbf936390e3c02e64c7ae94eb7d8066962df93b"}
Sep 16 12:49:04.015361 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.015345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:49:04.030384 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.030340    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bj87h" podStartSLOduration=66.091623267 podStartE2EDuration="1m11.030327435s" totalImagesPullingTime="4.938704168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:04.029260971 +0000 UTC m=+71.979606521" watchObservedRunningTime="2026-09-16 12:49:04.030327435 +0000 UTC m=+71.980673009"
Sep 16 12:49:04.047869 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.047816    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fr8w4" podStartSLOduration=66.428812006 podStartE2EDuration="1m11.047805449s" totalImagesPullingTime="4.618993443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:04.046288897 +0000 UTC m=+71.996634450" watchObservedRunningTime="2026-09-16 12:49:04.047805449 +0000 UTC m=+71.998151001"
Sep 16 12:49:04.133195 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.133175    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zmc7\""
Sep 16 12:49:04.141376 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.141356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tjwqw"
Sep 16 12:49:04.428983 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.428951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tjwqw"]
Sep 16 12:49:04.432785 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:04.432756    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381d5120_3a5b_43b1_af75_56abb87e27b5.slice/crio-b52da41a4f2d80b70a8e945845826e3f0bf09cb1dfe50176c0610ed4ba0755ac WatchSource:0}: Error finding container b52da41a4f2d80b70a8e945845826e3f0bf09cb1dfe50176c0610ed4ba0755ac: Status 404 returned error can't find the container with id b52da41a4f2d80b70a8e945845826e3f0bf09cb1dfe50176c0610ed4ba0755ac
Sep 16 12:49:04.567191 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:04.567166    2577 scope.go:117] "RemoveContainer" containerID="2d337e2734f2c0afe378cf513b58205634fa4be3066bc3129daee0a9494a13af"
Sep 16 12:49:05.020870 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.020825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tjwqw" event={"ID":"381d5120-3a5b-43b1-af75-56abb87e27b5","Type":"ContainerStarted","Data":"1aa604cbf45db4881c207fbaf424632d5a1de36c26be41c8e89a1e4f644da673"}
Sep 16 12:49:05.021294 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.020901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tjwqw" event={"ID":"381d5120-3a5b-43b1-af75-56abb87e27b5","Type":"ContainerStarted","Data":"b52da41a4f2d80b70a8e945845826e3f0bf09cb1dfe50176c0610ed4ba0755ac"}
Sep 16 12:49:05.022639 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.022620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:49:05.022743 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.022724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" event={"ID":"088a59c6-e72f-409f-b775-369532584ed0","Type":"ContainerStarted","Data":"6cd45d2a90c4fcdd5bc91be8d24de073bc808afdffc7006fbe00dab6144b562e"}
Sep 16 12:49:05.023122 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.023098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:49:05.051790 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.051743    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm" podStartSLOduration=20.549348141 podStartE2EDuration="24.051725327s" totalImagesPullingTime="3.502377186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:05.051376636 +0000 UTC m=+73.001722191" watchObservedRunningTime="2026-09-16 12:49:05.051725327 +0000 UTC m=+73.002070885"
Sep 16 12:49:05.248014 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:05.247985    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-mbvzm"
Sep 16 12:49:06.029930 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:06.029893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tjwqw" event={"ID":"381d5120-3a5b-43b1-af75-56abb87e27b5","Type":"ContainerStarted","Data":"cd8f7d62dcb2a733a634c933146bd554c1a22559caaca125896cbb8dbfad0478"}
Sep 16 12:49:07.035394 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:07.035358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tjwqw" event={"ID":"381d5120-3a5b-43b1-af75-56abb87e27b5","Type":"ContainerStarted","Data":"54ef974a97389ced7b0229551e94cb7b9a1847f562f0e8e7fa60449bc0e8fa21"}
Sep 16 12:49:07.059290 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:07.059239    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tjwqw" podStartSLOduration=19.321021251 podStartE2EDuration="21.059221195s" totalImagesPullingTime="1.738199944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:07.054295752 +0000 UTC m=+75.004641302" watchObservedRunningTime="2026-09-16 12:49:07.059221195 +0000 UTC m=+75.009566747"
Sep 16 12:49:10.018023 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:10.017996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l9s7f"
Sep 16 12:49:13.027309 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.027274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74dfb48679-kcvt2"]
Sep 16 12:49:13.031886 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.031840    2577 patch_prober.go:28] interesting pod/image-registry-74dfb48679-kcvt2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:49:13.032027 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.031915    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" podUID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:49:13.083433 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.083400    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-pddqc"]
Sep 16 12:49:13.085367 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.085347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:13.089046 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.089028    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:49:13.089270 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.089243    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-z6phm\""
Sep 16 12:49:13.089471 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.089457    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:49:13.113009 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.112989    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-pddqc"]
Sep 16 12:49:13.187845 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.187819    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55685f76cf-92pmk"]
Sep 16 12:49:13.189726 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.189711    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.194972 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.194948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hzs\" (UniqueName: \"kubernetes.io/projected/ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa-kube-api-access-b2hzs\") pod \"downloads-76f99584f4-pddqc\" (UID: \"ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa\") " pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:13.224185 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.224163    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55685f76cf-92pmk"]
Sep 16 12:49:13.296298 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb285905-617d-4e32-870a-9ba9b7c6380d-registry-certificates\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296298 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb285905-617d-4e32-870a-9ba9b7c6380d-installation-pull-secrets\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296298 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb285905-617d-4e32-870a-9ba9b7c6380d-ca-trust-extracted\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296478 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb285905-617d-4e32-870a-9ba9b7c6380d-image-registry-private-configuration\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296478 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-registry-tls\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296478 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hzs\" (UniqueName: \"kubernetes.io/projected/ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa-kube-api-access-b2hzs\") pod \"downloads-76f99584f4-pddqc\" (UID: \"ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa\") " pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:13.296576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-bound-sa-token\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2ws\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-kube-api-access-kn2ws\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.296576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.296536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb285905-617d-4e32-870a-9ba9b7c6380d-trusted-ca\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.313676 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.313657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hzs\" (UniqueName: \"kubernetes.io/projected/ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa-kube-api-access-b2hzs\") pod \"downloads-76f99584f4-pddqc\" (UID: \"ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa\") " pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:13.393901 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.393874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:13.397707 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb285905-617d-4e32-870a-9ba9b7c6380d-registry-certificates\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.397813 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb285905-617d-4e32-870a-9ba9b7c6380d-installation-pull-secrets\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.397813 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb285905-617d-4e32-870a-9ba9b7c6380d-ca-trust-extracted\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.397813 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb285905-617d-4e32-870a-9ba9b7c6380d-image-registry-private-configuration\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.397813 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-registry-tls\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.398044 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-bound-sa-token\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.398044 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2ws\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-kube-api-access-kn2ws\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.398044 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.397923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb285905-617d-4e32-870a-9ba9b7c6380d-trusted-ca\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.398279 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.398252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb285905-617d-4e32-870a-9ba9b7c6380d-ca-trust-extracted\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.398794 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.398772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb285905-617d-4e32-870a-9ba9b7c6380d-trusted-ca\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.400467 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.400441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb285905-617d-4e32-870a-9ba9b7c6380d-registry-certificates\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.400665 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.400644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb285905-617d-4e32-870a-9ba9b7c6380d-installation-pull-secrets\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.401097 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.401075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb285905-617d-4e32-870a-9ba9b7c6380d-image-registry-private-configuration\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.402617 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.402596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-registry-tls\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.413201 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.413176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2ws\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-kube-api-access-kn2ws\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.416996 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.416971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb285905-617d-4e32-870a-9ba9b7c6380d-bound-sa-token\") pod \"image-registry-55685f76cf-92pmk\" (UID: \"cb285905-617d-4e32-870a-9ba9b7c6380d\") " pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.498038 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.498007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:13.598968 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.598900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:49:13.601220 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.601197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f68d64d-28fc-411d-85ec-8fc331c3e756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfhlf\" (UID: \"4f68d64d-28fc-411d-85ec-8fc331c3e756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:49:13.683899 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.683792    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-pddqc"]
Sep 16 12:49:13.686586 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:13.686563    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9d7c0c_ba72_40ae_a0a6_6c2009ff74aa.slice/crio-9b969e3738c962cb6e48a70c911c31263158a7886abbe4c648ef11ff2e833b9d WatchSource:0}: Error finding container 9b969e3738c962cb6e48a70c911c31263158a7886abbe4c648ef11ff2e833b9d: Status 404 returned error can't find the container with id 9b969e3738c962cb6e48a70c911c31263158a7886abbe4c648ef11ff2e833b9d
Sep 16 12:49:13.795780 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.795747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55685f76cf-92pmk"]
Sep 16 12:49:13.798625 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:13.798601    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb285905_617d_4e32_870a_9ba9b7c6380d.slice/crio-24c8846fb64b35f7b8cbd7df67404f10e9a4957601bf39421af995af701476f7 WatchSource:0}: Error finding container 24c8846fb64b35f7b8cbd7df67404f10e9a4957601bf39421af995af701476f7: Status 404 returned error can't find the container with id 24c8846fb64b35f7b8cbd7df67404f10e9a4957601bf39421af995af701476f7
Sep 16 12:49:13.800654 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.800636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xqknp\""
Sep 16 12:49:13.803342 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:13.803321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"
Sep 16 12:49:14.059140 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:14.059097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55685f76cf-92pmk" event={"ID":"cb285905-617d-4e32-870a-9ba9b7c6380d","Type":"ContainerStarted","Data":"907f0678265d1c9c529ae4a5d468c925047d94332e738b922c334adb94122694"}
Sep 16 12:49:14.059566 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:14.059159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55685f76cf-92pmk" event={"ID":"cb285905-617d-4e32-870a-9ba9b7c6380d","Type":"ContainerStarted","Data":"24c8846fb64b35f7b8cbd7df67404f10e9a4957601bf39421af995af701476f7"}
Sep 16 12:49:14.059566 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:14.059234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:14.060482 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:14.060462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-pddqc" event={"ID":"ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa","Type":"ContainerStarted","Data":"9b969e3738c962cb6e48a70c911c31263158a7886abbe4c648ef11ff2e833b9d"}
Sep 16 12:49:14.094721 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:14.094698    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf"]
Sep 16 12:49:14.095780 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:14.095758    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f68d64d_28fc_411d_85ec_8fc331c3e756.slice/crio-5222e8f2d5077ae786ddb05bb6f81e45673e8b29fd193178226b1668183d29d6 WatchSource:0}: Error finding container 5222e8f2d5077ae786ddb05bb6f81e45673e8b29fd193178226b1668183d29d6: Status 404 returned error can't find the container with id 5222e8f2d5077ae786ddb05bb6f81e45673e8b29fd193178226b1668183d29d6
Sep 16 12:49:15.067580 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:15.067536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf" event={"ID":"4f68d64d-28fc-411d-85ec-8fc331c3e756","Type":"ContainerStarted","Data":"5222e8f2d5077ae786ddb05bb6f81e45673e8b29fd193178226b1668183d29d6"}
Sep 16 12:49:17.078599 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:17.078550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf" event={"ID":"4f68d64d-28fc-411d-85ec-8fc331c3e756","Type":"ContainerStarted","Data":"77092aa21352cef906e9fbff31261af3225d1158aeacff71248a8430fa70b03a"}
Sep 16 12:49:17.104953 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:17.104900    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfhlf" podStartSLOduration=33.905615081 podStartE2EDuration="36.10488226s" totalImagesPullingTime="2.199267179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:17.104040123 +0000 UTC m=+85.054385696" watchObservedRunningTime="2026-09-16 12:49:17.10488226 +0000 UTC m=+85.055227814"
Sep 16 12:49:17.105426 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:17.105381    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55685f76cf-92pmk" podStartSLOduration=4.105368154 podStartE2EDuration="4.105368154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:14.095249173 +0000 UTC m=+82.045594726" watchObservedRunningTime="2026-09-16 12:49:17.105368154 +0000 UTC m=+85.055713706"
Sep 16 12:49:22.014495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.014458    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"]
Sep 16 12:49:22.017605 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.017578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.020485 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.020457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5wxbr\""
Sep 16 12:49:22.020634 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.020526    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 12:49:22.021390 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.021368    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 12:49:22.021555 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.021538    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:49:22.030738 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.030715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"]
Sep 16 12:49:22.177577 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.177546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.177741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.177598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cabd51-e2f9-4770-8228-0a83e785cd78-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.177741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.177629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8xs\" (UniqueName: \"kubernetes.io/projected/43cabd51-e2f9-4770-8228-0a83e785cd78-kube-api-access-8f8xs\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.177741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.177712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.278666 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.278585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.278666 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.278645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cabd51-e2f9-4770-8228-0a83e785cd78-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.278666 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.278668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8xs\" (UniqueName: \"kubernetes.io/projected/43cabd51-e2f9-4770-8228-0a83e785cd78-kube-api-access-8f8xs\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.278959 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.278716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.278959 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:22.278767    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 12:49:22.278959 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:22.278837    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-tls podName:43cabd51-e2f9-4770-8228-0a83e785cd78 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:22.778816789 +0000 UTC m=+90.729162328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-kshmf" (UID: "43cabd51-e2f9-4770-8228-0a83e785cd78") : secret "prometheus-operator-tls" not found
Sep 16 12:49:22.279768 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.279744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cabd51-e2f9-4770-8228-0a83e785cd78-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.281633 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.281612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.291620 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.291600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8xs\" (UniqueName: \"kubernetes.io/projected/43cabd51-e2f9-4770-8228-0a83e785cd78-kube-api-access-8f8xs\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.782363 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.782327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.785284 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.785259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43cabd51-e2f9-4770-8228-0a83e785cd78-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kshmf\" (UID: \"43cabd51-e2f9-4770-8228-0a83e785cd78\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:22.927427 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:22.927391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"
Sep 16 12:49:23.032486 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:23.032405    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:49:30.017349 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:30.017318    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kshmf"]
Sep 16 12:49:30.020117 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:30.020086    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cabd51_e2f9_4770_8228_0a83e785cd78.slice/crio-0465c6960a557c7be36512d003454a8fe527c72b552e23b4e0056d5e225591c1 WatchSource:0}: Error finding container 0465c6960a557c7be36512d003454a8fe527c72b552e23b4e0056d5e225591c1: Status 404 returned error can't find the container with id 0465c6960a557c7be36512d003454a8fe527c72b552e23b4e0056d5e225591c1
Sep 16 12:49:30.131068 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:30.131032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf" event={"ID":"43cabd51-e2f9-4770-8228-0a83e785cd78","Type":"ContainerStarted","Data":"0465c6960a557c7be36512d003454a8fe527c72b552e23b4e0056d5e225591c1"}
Sep 16 12:49:30.133080 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:30.133047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-pddqc" event={"ID":"ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa","Type":"ContainerStarted","Data":"2a03df51e488303031705110bc7800a79d1af33276bd98d0861aa48d75582a6f"}
Sep 16 12:49:30.133947 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:30.133924    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:30.148186 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:30.148162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-pddqc"
Sep 16 12:49:30.154178 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:30.154116    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-pddqc" podStartSLOduration=1.001712584 podStartE2EDuration="17.154094772s" totalImagesPullingTime="16.152382188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:30.153188307 +0000 UTC m=+98.103533864" watchObservedRunningTime="2026-09-16 12:49:30.154094772 +0000 UTC m=+98.104440329"
Sep 16 12:49:33.149535 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:33.149486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf" event={"ID":"43cabd51-e2f9-4770-8228-0a83e785cd78","Type":"ContainerStarted","Data":"50c3b71d8cb489596b449b08922cbf2cad630de6013ebca965490ba4c325379d"}
Sep 16 12:49:33.150159 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:33.149590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf" event={"ID":"43cabd51-e2f9-4770-8228-0a83e785cd78","Type":"ContainerStarted","Data":"4f4a9fff99508a5e9bf46a6e1c60074ff28b4d8ada8d278316096770e083d773"}
Sep 16 12:49:33.173958 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:33.173900    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kshmf" podStartSLOduration=10.125477672 podStartE2EDuration="12.173883667s" totalImagesPullingTime="2.048405995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:33.170770978 +0000 UTC m=+101.121116542" watchObservedRunningTime="2026-09-16 12:49:33.173883667 +0000 UTC m=+101.124229210"
Sep 16 12:49:35.026275 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.026239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fr8w4"
Sep 16 12:49:35.074313 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.074285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55685f76cf-92pmk"
Sep 16 12:49:35.561890 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.561838    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tzkpg"]
Sep 16 12:49:35.597353 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.597322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.601691 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.601457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:49:35.601691 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.601484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xr86w\""
Sep 16 12:49:35.601691 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.601663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:49:35.602773 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.602753    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:49:35.698234 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-sys\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.698473 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.698579 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e638f8-6925-4822-a940-b9318f0d13bd-metrics-client-ca\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.698739 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-root\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.698959 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-textfile\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.699064 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.699064 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.698996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-wtmp\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.699064 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.699022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4kkk\" (UniqueName: \"kubernetes.io/projected/72e638f8-6925-4822-a940-b9318f0d13bd-kube-api-access-h4kkk\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.699219 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.699069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800040 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.799996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-sys\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e638f8-6925-4822-a940-b9318f0d13bd-metrics-client-ca\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-root\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-textfile\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-wtmp\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800257 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4kkk\" (UniqueName: \"kubernetes.io/projected/72e638f8-6925-4822-a940-b9318f0d13bd-kube-api-access-h4kkk\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800618 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.800935 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.801030 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.800992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-sys\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.801588 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.801563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-textfile\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.801762 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.801746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-root\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.801881 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.801793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72e638f8-6925-4822-a940-b9318f0d13bd-metrics-client-ca\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.801984 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.801925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-wtmp\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.802067 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:35.802002    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 12:49:35.802249 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:35.802237    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls podName:72e638f8-6925-4822-a940-b9318f0d13bd nodeName:}" failed. No retries permitted until 2026-09-16 12:49:36.302213407 +0000 UTC m=+104.252558957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls") pod "node-exporter-tzkpg" (UID: "72e638f8-6925-4822-a940-b9318f0d13bd") : secret "node-exporter-tls" not found
Sep 16 12:49:35.803757 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.803732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:35.814421 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:35.814368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4kkk\" (UniqueName: \"kubernetes.io/projected/72e638f8-6925-4822-a940-b9318f0d13bd-kube-api-access-h4kkk\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:36.305154 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:36.305099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:36.305701 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:36.305280    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 12:49:36.305701 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:36.305353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls podName:72e638f8-6925-4822-a940-b9318f0d13bd nodeName:}" failed. No retries permitted until 2026-09-16 12:49:37.305333154 +0000 UTC m=+105.255678704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls") pod "node-exporter-tzkpg" (UID: "72e638f8-6925-4822-a940-b9318f0d13bd") : secret "node-exporter-tls" not found
Sep 16 12:49:37.314741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.314701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:37.317622 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.317602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/72e638f8-6925-4822-a940-b9318f0d13bd-node-exporter-tls\") pod \"node-exporter-tzkpg\" (UID: \"72e638f8-6925-4822-a940-b9318f0d13bd\") " pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:37.412900 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.412837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tzkpg"
Sep 16 12:49:37.445798 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:37.445767    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e638f8_6925_4822_a940_b9318f0d13bd.slice/crio-3704f0fac43cc4c8764e0624b96d851357440df42f3596161b4633724777831a WatchSource:0}: Error finding container 3704f0fac43cc4c8764e0624b96d851357440df42f3596161b4633724777831a: Status 404 returned error can't find the container with id 3704f0fac43cc4c8764e0624b96d851357440df42f3596161b4633724777831a
Sep 16 12:49:37.520370 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.520336    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bdfd595d9-thbtv"]
Sep 16 12:49:37.554937 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.554910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bdfd595d9-thbtv"]
Sep 16 12:49:37.555088 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.555058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.558398 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.558375    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7ikml1dm0gr8s\""
Sep 16 12:49:37.558786 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.558755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 12:49:37.558911 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.558882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 12:49:37.558982 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.558919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 12:49:37.559180 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.559162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 12:49:37.559347 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.559329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 12:49:37.559545 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.559390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9qkjv\""
Sep 16 12:49:37.617870 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.617786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f57d20f-7e64-4959-aa61-61537eb7bd55-metrics-client-ca\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618032 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-grpc-tls\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618101 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618157 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vtt\" (UniqueName: \"kubernetes.io/projected/0f57d20f-7e64-4959-aa61-61537eb7bd55-kube-api-access-j7vtt\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618210 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618210 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-tls\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618311 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.618350 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.618315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.719581 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f57d20f-7e64-4959-aa61-61537eb7bd55-metrics-client-ca\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.719581 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-grpc-tls\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.720187 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.720187 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vtt\" (UniqueName: \"kubernetes.io/projected/0f57d20f-7e64-4959-aa61-61537eb7bd55-kube-api-access-j7vtt\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.720187 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.720187 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-tls\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.720187 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.720187 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.719967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.721043 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.720393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f57d20f-7e64-4959-aa61-61537eb7bd55-metrics-client-ca\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.723153 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.723126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.723466 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.723443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-tls\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.723563 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.723445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-grpc-tls\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.724133 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.724108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.724590 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.724561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.725531 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.725510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0f57d20f-7e64-4959-aa61-61537eb7bd55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.728288 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.728268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vtt\" (UniqueName: \"kubernetes.io/projected/0f57d20f-7e64-4959-aa61-61537eb7bd55-kube-api-access-j7vtt\") pod \"thanos-querier-bdfd595d9-thbtv\" (UID: \"0f57d20f-7e64-4959-aa61-61537eb7bd55\") " pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:37.867771 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:37.867735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:38.047184 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.047145    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" podUID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" containerName="registry" containerID="cri-o://a8ad68bb17ddf189d2270148857b02d7f64c4118e97a1ef81880ac349538614c" gracePeriod=30
Sep 16 12:49:38.173992 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.173917    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" containerID="a8ad68bb17ddf189d2270148857b02d7f64c4118e97a1ef81880ac349538614c" exitCode=0
Sep 16 12:49:38.174143 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.173995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" event={"ID":"f0ba488e-ff53-4078-a1e6-a65634cc90b3","Type":"ContainerDied","Data":"a8ad68bb17ddf189d2270148857b02d7f64c4118e97a1ef81880ac349538614c"}
Sep 16 12:49:38.175565 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.175544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzkpg" event={"ID":"72e638f8-6925-4822-a940-b9318f0d13bd","Type":"ContainerStarted","Data":"3704f0fac43cc4c8764e0624b96d851357440df42f3596161b4633724777831a"}
Sep 16 12:49:38.192908 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.192880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bdfd595d9-thbtv"]
Sep 16 12:49:38.194958 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:38.194685    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f57d20f_7e64_4959_aa61_61537eb7bd55.slice/crio-95fb758603a395b5aa0c6f8cb9b966c11abadd17b3462da2c873a1292918c08e WatchSource:0}: Error finding container 95fb758603a395b5aa0c6f8cb9b966c11abadd17b3462da2c873a1292918c08e: Status 404 returned error can't find the container with id 95fb758603a395b5aa0c6f8cb9b966c11abadd17b3462da2c873a1292918c08e
Sep 16 12:49:38.340474 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.340386    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0ba488e-ff53-4078-a1e6-a65634cc90b3-ca-trust-extracted\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-image-registry-private-configuration\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7hd\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-kube-api-access-st7hd\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-trusted-ca\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-certificates\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-installation-pull-secrets\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-bound-sa-token\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.427761 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.427715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") pod \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\" (UID: \"f0ba488e-ff53-4078-a1e6-a65634cc90b3\") "
Sep 16 12:49:38.428730 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.428691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:49:38.429810 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.429770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:49:38.432295 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.432267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:49:38.433601 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.433572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-kube-api-access-st7hd" (OuterVolumeSpecName: "kube-api-access-st7hd") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "kube-api-access-st7hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:38.434239 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.434212    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:49:38.435538 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.435503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:38.440148 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.439563    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ba488e-ff53-4078-a1e6-a65634cc90b3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:49:38.442672 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.441851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f0ba488e-ff53-4078-a1e6-a65634cc90b3" (UID: "f0ba488e-ff53-4078-a1e6-a65634cc90b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:38.529475 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529430    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0ba488e-ff53-4078-a1e6-a65634cc90b3-ca-trust-extracted\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529475 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529461    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-image-registry-private-configuration\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529475 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529475    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7hd\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-kube-api-access-st7hd\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529489    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-trusted-ca\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529497    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-certificates\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529506    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0ba488e-ff53-4078-a1e6-a65634cc90b3-installation-pull-secrets\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529514    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-bound-sa-token\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:38.529741 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:38.529524    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0ba488e-ff53-4078-a1e6-a65634cc90b3-registry-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:49:39.182619 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.182577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2" event={"ID":"f0ba488e-ff53-4078-a1e6-a65634cc90b3","Type":"ContainerDied","Data":"51c056ef9a41d04b0af9a67cc67bab81effd40bd81cd10406d6c549505b381b5"}
Sep 16 12:49:39.182801 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.182646    2577 scope.go:117] "RemoveContainer" containerID="a8ad68bb17ddf189d2270148857b02d7f64c4118e97a1ef81880ac349538614c"
Sep 16 12:49:39.182892 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.182813    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dfb48679-kcvt2"
Sep 16 12:49:39.185076 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.185046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"95fb758603a395b5aa0c6f8cb9b966c11abadd17b3462da2c873a1292918c08e"}
Sep 16 12:49:39.219592 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.219566    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74dfb48679-kcvt2"]
Sep 16 12:49:39.224628 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.224543    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74dfb48679-kcvt2"]
Sep 16 12:49:39.965271 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.965234    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75f78dbf54-4bw5b"]
Sep 16 12:49:39.965708 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.965682    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" containerName="registry"
Sep 16 12:49:39.965708 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.965700    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" containerName="registry"
Sep 16 12:49:39.965801 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.965758    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" containerName="registry"
Sep 16 12:49:39.986144 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.986116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f78dbf54-4bw5b"]
Sep 16 12:49:39.986308 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.986222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:39.990837 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.990428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 12:49:39.990837 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.990461    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 12:49:39.990837 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.990579    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c0us7jhbhaleq\""
Sep 16 12:49:39.990837 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.990657    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7w6cn\""
Sep 16 12:49:39.990837 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.990769    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:49:39.991208 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:39.990898    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 12:49:40.042387 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-secret-metrics-server-tls\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.042575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac2ce0be-e36a-4f99-9884-b5d6a233796c-metrics-server-audit-profiles\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.042575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-secret-metrics-server-client-certs\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.042575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac2ce0be-e36a-4f99-9884-b5d6a233796c-audit-log\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.042575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-client-ca-bundle\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.042575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvh5k\" (UniqueName: \"kubernetes.io/projected/ac2ce0be-e36a-4f99-9884-b5d6a233796c-kube-api-access-dvh5k\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.042786 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.042610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac2ce0be-e36a-4f99-9884-b5d6a233796c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.143803 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.143708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-secret-metrics-server-client-certs\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.143803 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.143747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac2ce0be-e36a-4f99-9884-b5d6a233796c-audit-log\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.143803 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.143774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-client-ca-bundle\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.143803 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.143801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvh5k\" (UniqueName: \"kubernetes.io/projected/ac2ce0be-e36a-4f99-9884-b5d6a233796c-kube-api-access-dvh5k\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.144154 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.143888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac2ce0be-e36a-4f99-9884-b5d6a233796c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.144226 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.144192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac2ce0be-e36a-4f99-9884-b5d6a233796c-audit-log\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.144309 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.144226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-secret-metrics-server-tls\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.144309 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.144262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac2ce0be-e36a-4f99-9884-b5d6a233796c-metrics-server-audit-profiles\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.144608 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.144556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac2ce0be-e36a-4f99-9884-b5d6a233796c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.145321 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.145297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac2ce0be-e36a-4f99-9884-b5d6a233796c-metrics-server-audit-profiles\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.146964 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.146920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-client-ca-bundle\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.148679 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.148656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-secret-metrics-server-client-certs\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.151423 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.151400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac2ce0be-e36a-4f99-9884-b5d6a233796c-secret-metrics-server-tls\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.160459 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.160429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvh5k\" (UniqueName: \"kubernetes.io/projected/ac2ce0be-e36a-4f99-9884-b5d6a233796c-kube-api-access-dvh5k\") pod \"metrics-server-75f78dbf54-4bw5b\" (UID: \"ac2ce0be-e36a-4f99-9884-b5d6a233796c\") " pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.179418 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.179389    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"]
Sep 16 12:49:40.196016 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.195988    2577 generic.go:356] "Generic (PLEG): container finished" podID="72e638f8-6925-4822-a940-b9318f0d13bd" containerID="fa701a7f90bfa767865e9a6b0b60b5b92f03b1b9bb41a519f6b932554fd78985" exitCode=0
Sep 16 12:49:40.219901 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.219819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"]
Sep 16 12:49:40.219901 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.219890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzkpg" event={"ID":"72e638f8-6925-4822-a940-b9318f0d13bd","Type":"ContainerDied","Data":"fa701a7f90bfa767865e9a6b0b60b5b92f03b1b9bb41a519f6b932554fd78985"}
Sep 16 12:49:40.220067 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.219967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.223892 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.223663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kfgkf\""
Sep 16 12:49:40.223892 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.223705    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 12:49:40.227165 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.227146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56b4c467b-xhthc"]
Sep 16 12:49:40.246191 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.246167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.246329 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.246297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl64f\" (UniqueName: \"kubernetes.io/projected/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-kube-api-access-dl64f\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.248939 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.248913    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4c467b-xhthc"]
Sep 16 12:49:40.249066 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.249043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.255462 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.255442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:49:40.256066 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.255627    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:49:40.256220 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.256198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:49:40.256430 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.256388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:49:40.256833 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.256811    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:49:40.257014 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.256996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d5xsr\""
Sep 16 12:49:40.268965 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.268947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:49:40.296943 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.296918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:49:40.347411 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-oauth-config\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347535 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczs6\" (UniqueName: \"kubernetes.io/projected/8cea2fb1-54d4-4140-a019-9c3f36396d08-kube-api-access-rczs6\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347535 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-trusted-ca-bundle\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347535 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl64f\" (UniqueName: \"kubernetes.io/projected/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-kube-api-access-dl64f\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.347535 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-serving-cert\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347700 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-service-ca\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347700 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-oauth-serving-cert\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347700 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.347700 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.347625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-config\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.347834 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:40.347742    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 12:49:40.347834 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:49:40.347783    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-monitoring-plugin-cert podName:776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d nodeName:}" failed. No retries permitted until 2026-09-16 12:49:40.847768815 +0000 UTC m=+108.798114345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-l7sl9" (UID: "776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d") : secret "monitoring-plugin-cert" not found
Sep 16 12:49:40.363695 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.363663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl64f\" (UniqueName: \"kubernetes.io/projected/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-kube-api-access-dl64f\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.448401 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-oauth-config\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.448480 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczs6\" (UniqueName: \"kubernetes.io/projected/8cea2fb1-54d4-4140-a019-9c3f36396d08-kube-api-access-rczs6\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.448480 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-trusted-ca-bundle\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.448585 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-serving-cert\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.448585 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-service-ca\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.448585 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-oauth-serving-cert\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.448728 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.448673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-config\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.449530 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.449487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-oauth-serving-cert\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.449627 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.449512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-service-ca\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.449711 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.449682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-config\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.449818 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.449717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-trusted-ca-bundle\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.451206 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.451187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-serving-cert\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.451299 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.451272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-oauth-config\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.459228 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.459207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczs6\" (UniqueName: \"kubernetes.io/projected/8cea2fb1-54d4-4140-a019-9c3f36396d08-kube-api-access-rczs6\") pod \"console-56b4c467b-xhthc\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") " pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.559517 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.559492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:49:40.574883 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.572963    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ba488e-ff53-4078-a1e6-a65634cc90b3" path="/var/lib/kubelet/pods/f0ba488e-ff53-4078-a1e6-a65634cc90b3/volumes"
Sep 16 12:49:40.594064 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.594039    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f78dbf54-4bw5b"]
Sep 16 12:49:40.644375 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:40.644343    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2ce0be_e36a_4f99_9884_b5d6a233796c.slice/crio-9c1a5153069cb78c203a03928a0fe2c60981db74b1feaca6fff326a7f55d7074 WatchSource:0}: Error finding container 9c1a5153069cb78c203a03928a0fe2c60981db74b1feaca6fff326a7f55d7074: Status 404 returned error can't find the container with id 9c1a5153069cb78c203a03928a0fe2c60981db74b1feaca6fff326a7f55d7074
Sep 16 12:49:40.731586 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.731527    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69b68dc588-p9mgt"]
Sep 16 12:49:40.758196 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.758167    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69b68dc588-p9mgt"]
Sep 16 12:49:40.758304 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.758270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.761113 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.761094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 12:49:40.761113 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.761106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 12:49:40.761280 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.761257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 12:49:40.761538 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.761500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wcfsd\""
Sep 16 12:49:40.761674 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.761558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 12:49:40.761674 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.761503    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 12:49:40.767224 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.767205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 12:49:40.851882 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.851835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-secret-telemeter-client\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.851978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.851937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.851978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.851970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-serving-certs-ca-bundle\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.852049 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.852000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9q6\" (UniqueName: \"kubernetes.io/projected/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-kube-api-access-qk9q6\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.852049 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.852020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-federate-client-tls\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.852049 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.852040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-telemeter-client-tls\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.852141 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.852070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-metrics-client-ca\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.852141 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.852085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.852141 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.852125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.854492 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.854472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-l7sl9\" (UID: \"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:40.950041 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.950021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4c467b-xhthc"]
Sep 16 12:49:40.951930 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:40.951902    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cea2fb1_54d4_4140_a019_9c3f36396d08.slice/crio-3e1fa89ce979fb5bc7961af9a48e16b5de61a7df300598853514a7b25c6749d0 WatchSource:0}: Error finding container 3e1fa89ce979fb5bc7961af9a48e16b5de61a7df300598853514a7b25c6749d0: Status 404 returned error can't find the container with id 3e1fa89ce979fb5bc7961af9a48e16b5de61a7df300598853514a7b25c6749d0
Sep 16 12:49:40.953115 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953181 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-serving-certs-ca-bundle\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953181 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9q6\" (UniqueName: \"kubernetes.io/projected/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-kube-api-access-qk9q6\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953271 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-federate-client-tls\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953271 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-telemeter-client-tls\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953271 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-metrics-client-ca\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953410 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.953410 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.953355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-secret-telemeter-client\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.954794 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.954148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-serving-certs-ca-bundle\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.954794 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.954332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-metrics-client-ca\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.954794 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.954771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.956558 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.956528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-federate-client-tls\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.956750 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.956730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-telemeter-client-tls\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.956819 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.956780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.957119 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.957099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-secret-telemeter-client\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:40.962326 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:40.962307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9q6\" (UniqueName: \"kubernetes.io/projected/216f4d9d-2aad-43a1-aa3a-4fe6ba83a096-kube-api-access-qk9q6\") pod \"telemeter-client-69b68dc588-p9mgt\" (UID: \"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096\") " pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:41.068297 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.068269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt"
Sep 16 12:49:41.132745 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.132465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:41.205932 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.205899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"50f334749620a95762338246589d211018bbb341c525bb0469f311159e4e6a68"}
Sep 16 12:49:41.206040 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.205984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"39a933cf1c249e700a8abbd43c72050d1d814cdda2bc52e7c5eb08272e2b17d7"}
Sep 16 12:49:41.206040 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.206025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"b1c3f37a5b2a573947d68ad68b8909a2db9cb4c778f970798841a2f6194aaaa0"}
Sep 16 12:49:41.209301 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.209271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4c467b-xhthc" event={"ID":"8cea2fb1-54d4-4140-a019-9c3f36396d08","Type":"ContainerStarted","Data":"3e1fa89ce979fb5bc7961af9a48e16b5de61a7df300598853514a7b25c6749d0"}
Sep 16 12:49:41.213203 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.213180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzkpg" event={"ID":"72e638f8-6925-4822-a940-b9318f0d13bd","Type":"ContainerStarted","Data":"32287910fc513169014e76438df847be9a1ca6ab71720ea68a6fac21b88cc0af"}
Sep 16 12:49:41.213307 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.213227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tzkpg" event={"ID":"72e638f8-6925-4822-a940-b9318f0d13bd","Type":"ContainerStarted","Data":"d7e4b6806ef95320aa45f2cbd5f34e9a453e7031e0c42044ab243cca53716acd"}
Sep 16 12:49:41.214832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.214804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b" event={"ID":"ac2ce0be-e36a-4f99-9884-b5d6a233796c","Type":"ContainerStarted","Data":"9c1a5153069cb78c203a03928a0fe2c60981db74b1feaca6fff326a7f55d7074"}
Sep 16 12:49:41.233930 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.233833    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tzkpg" podStartSLOduration=4.149496662 podStartE2EDuration="6.233818348s" totalImagesPullingTime="1.994935631s" totalInitContainerRuntime="89.386055ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:41.232070931 +0000 UTC m=+109.182416476" watchObservedRunningTime="2026-09-16 12:49:41.233818348 +0000 UTC m=+109.184163900"
Sep 16 12:49:41.359878 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.359833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69b68dc588-p9mgt"]
Sep 16 12:49:41.363519 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:41.363489    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f4d9d_2aad_43a1_aa3a_4fe6ba83a096.slice/crio-7bbf555907bb7840d501b5a40e2ae85d10f3ebffbbe926137639dff5990af917 WatchSource:0}: Error finding container 7bbf555907bb7840d501b5a40e2ae85d10f3ebffbbe926137639dff5990af917: Status 404 returned error can't find the container with id 7bbf555907bb7840d501b5a40e2ae85d10f3ebffbbe926137639dff5990af917
Sep 16 12:49:41.429152 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:41.428832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"]
Sep 16 12:49:41.431448 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:49:41.431421    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776bc35e_d960_4fe2_a6d0_5ab3c4f08c4d.slice/crio-a133debb97a6bf97b3266c47dbec09ab382907580a10b30187c7bf19edd47925 WatchSource:0}: Error finding container a133debb97a6bf97b3266c47dbec09ab382907580a10b30187c7bf19edd47925: Status 404 returned error can't find the container with id a133debb97a6bf97b3266c47dbec09ab382907580a10b30187c7bf19edd47925
Sep 16 12:49:42.220931 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:42.220893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9" event={"ID":"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d","Type":"ContainerStarted","Data":"a133debb97a6bf97b3266c47dbec09ab382907580a10b30187c7bf19edd47925"}
Sep 16 12:49:42.223198 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:42.223125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt" event={"ID":"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096","Type":"ContainerStarted","Data":"7bbf555907bb7840d501b5a40e2ae85d10f3ebffbbe926137639dff5990af917"}
Sep 16 12:49:43.229322 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:43.229242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b" event={"ID":"ac2ce0be-e36a-4f99-9884-b5d6a233796c","Type":"ContainerStarted","Data":"a293d0357bdd1f18adbe8a576cf8901fc3617fb83414f30d208f5a8301f976fe"}
Sep 16 12:49:43.232355 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:43.232328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"0e81651aa9f6e4cc7bc8d303853a4f87ebab2abb818103cbf9ebc9252b64fa4c"}
Sep 16 12:49:43.251178 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:43.251124    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b" podStartSLOduration=2.054475888 podStartE2EDuration="4.251109244s" totalImagesPullingTime="2.196633356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:43.248655519 +0000 UTC m=+111.199001072" watchObservedRunningTime="2026-09-16 12:49:43.251109244 +0000 UTC m=+111.201454796"
Sep 16 12:49:46.248113 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:46.248080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9" event={"ID":"776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d","Type":"ContainerStarted","Data":"c8394887e285b3e8e1da5c47546c03efccfff5a832708fc547e32bb2e8a0b4ad"}
Sep 16 12:49:46.250061 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:46.250028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt" event={"ID":"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096","Type":"ContainerStarted","Data":"f8736a6503240667761018416adb87affa2e990b8068155ac8f88f4739fa55ff"}
Sep 16 12:49:46.253845 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:46.253806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"787ce8f64b90bd3aab2445473bc66324dae773b1afad444c0a11f15226d51424"}
Sep 16 12:49:46.255838 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:46.255775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4c467b-xhthc" event={"ID":"8cea2fb1-54d4-4140-a019-9c3f36396d08","Type":"ContainerStarted","Data":"5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03"}
Sep 16 12:49:47.264252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:47.264196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" event={"ID":"0f57d20f-7e64-4959-aa61-61537eb7bd55","Type":"ContainerStarted","Data":"2343fa3d419d2135866e8cf66de2e85bff2db4470487252ac78fe39691d80ea8"}
Sep 16 12:49:47.264828 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:47.264438    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:47.269329 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:47.269301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9"
Sep 16 12:49:47.294984 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:47.294735    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv" podStartSLOduration=5.957632125 podStartE2EDuration="10.29471935s" totalImagesPullingTime="4.337087225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:47.290764373 +0000 UTC m=+115.241109926" watchObservedRunningTime="2026-09-16 12:49:47.29471935 +0000 UTC m=+115.245064904"
Sep 16 12:49:47.317397 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:47.317351    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-l7sl9" podStartSLOduration=2.676119911 podStartE2EDuration="7.317338059s" totalImagesPullingTime="4.641218148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:47.316321556 +0000 UTC m=+115.266667108" watchObservedRunningTime="2026-09-16 12:49:47.317338059 +0000 UTC m=+115.267683622"
Sep 16 12:49:47.359158 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:47.359106    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56b4c467b-xhthc" podStartSLOduration=2.234904123 podStartE2EDuration="7.359088297s" totalImagesPullingTime="5.124184174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:47.357992128 +0000 UTC m=+115.308337680" watchObservedRunningTime="2026-09-16 12:49:47.359088297 +0000 UTC m=+115.309433850"
Sep 16 12:49:48.270714 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:48.270677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt" event={"ID":"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096","Type":"ContainerStarted","Data":"f06adf63cc03db1f70550babb69c098333811a17eca10f66345dd359d13ee4f7"}
Sep 16 12:49:48.271098 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:48.270748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt" event={"ID":"216f4d9d-2aad-43a1-aa3a-4fe6ba83a096","Type":"ContainerStarted","Data":"afdc024fb9c69aaf09284678c1781742881a5dc6dbfd09e46ee63cb589289161"}
Sep 16 12:49:48.271098 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:48.271067    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:48.276901 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:48.276881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bdfd595d9-thbtv"
Sep 16 12:49:48.299166 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:48.299125    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69b68dc588-p9mgt" podStartSLOduration=2.1400489719999998 podStartE2EDuration="8.299110469s" totalImagesPullingTime="6.159061497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:48.298184012 +0000 UTC m=+116.248529577" watchObservedRunningTime="2026-09-16 12:49:48.299110469 +0000 UTC m=+116.249456022"
Sep 16 12:49:48.454805 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:48.454773    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4c467b-xhthc"]
Sep 16 12:49:50.560343 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:49:50.560297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:50:00.297454 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:00.297422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:50:00.297931 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:00.297482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:50:02.325482 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.325446    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ba4cd64-504f-4bae-9f8b-be8a24205fca" containerID="a1e90c0a0ad50e8af19f7354495a217852c547e235d726f19c6f0a3c8fcc7bf3" exitCode=0
Sep 16 12:50:02.325835 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.325522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jgggh" event={"ID":"4ba4cd64-504f-4bae-9f8b-be8a24205fca","Type":"ContainerDied","Data":"a1e90c0a0ad50e8af19f7354495a217852c547e235d726f19c6f0a3c8fcc7bf3"}
Sep 16 12:50:02.325901 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.325889    2577 scope.go:117] "RemoveContainer" containerID="a1e90c0a0ad50e8af19f7354495a217852c547e235d726f19c6f0a3c8fcc7bf3"
Sep 16 12:50:02.447627 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.447583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:50:02.455356 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.455329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb342f5-ba85-4bea-a1a5-fa75f526c896-metrics-certs\") pod \"network-metrics-daemon-mww2q\" (UID: \"8bb342f5-ba85-4bea-a1a5-fa75f526c896\") " pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:50:02.593174 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.593096    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9nrdj\""
Sep 16 12:50:02.600443 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.600421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mww2q"
Sep 16 12:50:02.885294 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:02.885227    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mww2q"]
Sep 16 12:50:02.888559 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:50:02.888532    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb342f5_ba85_4bea_a1a5_fa75f526c896.slice/crio-49826b16614badbfc7f088d1bda87ac2ab849dceffa8bffff6dcf36b110b11e4 WatchSource:0}: Error finding container 49826b16614badbfc7f088d1bda87ac2ab849dceffa8bffff6dcf36b110b11e4: Status 404 returned error can't find the container with id 49826b16614badbfc7f088d1bda87ac2ab849dceffa8bffff6dcf36b110b11e4
Sep 16 12:50:03.332021 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:03.331946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mww2q" event={"ID":"8bb342f5-ba85-4bea-a1a5-fa75f526c896","Type":"ContainerStarted","Data":"49826b16614badbfc7f088d1bda87ac2ab849dceffa8bffff6dcf36b110b11e4"}
Sep 16 12:50:03.334077 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:03.334048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jgggh" event={"ID":"4ba4cd64-504f-4bae-9f8b-be8a24205fca","Type":"ContainerStarted","Data":"8dff1287d181d838d4d445d480f0241dcc314f697f81f3e455e3a3086b058ae2"}
Sep 16 12:50:04.339517 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:04.339474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mww2q" event={"ID":"8bb342f5-ba85-4bea-a1a5-fa75f526c896","Type":"ContainerStarted","Data":"faa4084fc015b4f6538887862d9ed404bb51b2437049b1409e73042a7fd7170d"}
Sep 16 12:50:04.339517 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:04.339523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mww2q" event={"ID":"8bb342f5-ba85-4bea-a1a5-fa75f526c896","Type":"ContainerStarted","Data":"d0b1b8d2169020b41629cac606b99c11b13d36f244a79a426ed5e88134ac5e33"}
Sep 16 12:50:04.371442 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:04.371398    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mww2q" podStartSLOduration=130.339782367 podStartE2EDuration="2m11.371383347s" totalImagesPullingTime="1.03160098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:04.369900152 +0000 UTC m=+132.320245702" watchObservedRunningTime="2026-09-16 12:50:04.371383347 +0000 UTC m=+132.321728900"
Sep 16 12:50:07.352564 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:07.352529    2577 generic.go:356] "Generic (PLEG): container finished" podID="bef9cefd-33b8-413a-b33f-e158ac86980e" containerID="d8cbfd2cb7ae4be0fac12f0927ba92dcc43181ebcc5f38f51d37e5c91a679210" exitCode=0
Sep 16 12:50:07.352997 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:07.352596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9" event={"ID":"bef9cefd-33b8-413a-b33f-e158ac86980e","Type":"ContainerDied","Data":"d8cbfd2cb7ae4be0fac12f0927ba92dcc43181ebcc5f38f51d37e5c91a679210"}
Sep 16 12:50:07.352997 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:07.352895    2577 scope.go:117] "RemoveContainer" containerID="d8cbfd2cb7ae4be0fac12f0927ba92dcc43181ebcc5f38f51d37e5c91a679210"
Sep 16 12:50:08.359053 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:08.359022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gj6m9" event={"ID":"bef9cefd-33b8-413a-b33f-e158ac86980e","Type":"ContainerStarted","Data":"fc2956580edc0bb7296a7ad66d7aa5d2323e1d0e0545cb3c4ed98adb6ac8ab9b"}
Sep 16 12:50:12.376765 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:12.376733    2577 generic.go:356] "Generic (PLEG): container finished" podID="98d5b95e-0e6b-47a1-a36f-8604c6db1e0a" containerID="ef992bd10df1bd005349f2fdaa3c73b9af78de80a0f5b38328a6894f765ad326" exitCode=0
Sep 16 12:50:12.377147 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:12.376807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln" event={"ID":"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a","Type":"ContainerDied","Data":"ef992bd10df1bd005349f2fdaa3c73b9af78de80a0f5b38328a6894f765ad326"}
Sep 16 12:50:12.377188 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:12.377156    2577 scope.go:117] "RemoveContainer" containerID="ef992bd10df1bd005349f2fdaa3c73b9af78de80a0f5b38328a6894f765ad326"
Sep 16 12:50:13.383731 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:13.383689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-lc5ln" event={"ID":"98d5b95e-0e6b-47a1-a36f-8604c6db1e0a","Type":"ContainerStarted","Data":"f93d9ce230ba1fd6b3f1d04b64d75e68b9f413b533bc9777ca6c31cb8a62a96c"}
Sep 16 12:50:14.293466 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.293411    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56b4c467b-xhthc" podUID="8cea2fb1-54d4-4140-a019-9c3f36396d08" containerName="console" containerID="cri-o://5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03" gracePeriod=15
Sep 16 12:50:14.574976 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.574945    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:50:14.747885 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.747832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-trusted-ca-bundle\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748047 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.747918    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-service-ca\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748047 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.747947    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-oauth-serving-cert\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748047 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.747992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-serving-cert\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748047 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-config\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748276 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-oauth-config\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748276 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rczs6\" (UniqueName: \"kubernetes.io/projected/8cea2fb1-54d4-4140-a019-9c3f36396d08-kube-api-access-rczs6\") pod \"8cea2fb1-54d4-4140-a019-9c3f36396d08\" (UID: \"8cea2fb1-54d4-4140-a019-9c3f36396d08\") "
Sep 16 12:50:14.748375 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-service-ca" (OuterVolumeSpecName: "service-ca") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:14.748375 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:14.748375 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:14.748545 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.748409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-config" (OuterVolumeSpecName: "console-config") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:14.750437 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.750414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:14.750549 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.750457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:14.751097 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.750415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cea2fb1-54d4-4140-a019-9c3f36396d08-kube-api-access-rczs6" (OuterVolumeSpecName: "kube-api-access-rczs6") pod "8cea2fb1-54d4-4140-a019-9c3f36396d08" (UID: "8cea2fb1-54d4-4140-a019-9c3f36396d08"). InnerVolumeSpecName "kube-api-access-rczs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:14.849062 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.848999    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-oauth-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:14.849062 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.849024    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczs6\" (UniqueName: \"kubernetes.io/projected/8cea2fb1-54d4-4140-a019-9c3f36396d08-kube-api-access-rczs6\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:14.849062 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.849034    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-trusted-ca-bundle\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:14.849062 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.849043    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-service-ca\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:14.849062 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.849051    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-oauth-serving-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:14.849062 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.849059    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-serving-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:14.849342 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:14.849068    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cea2fb1-54d4-4140-a019-9c3f36396d08-console-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:50:15.393914 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.393851    2577 generic.go:356] "Generic (PLEG): container finished" podID="8cea2fb1-54d4-4140-a019-9c3f36396d08" containerID="5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03" exitCode=0
Sep 16 12:50:15.393914 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.393917    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4c467b-xhthc"
Sep 16 12:50:15.394152 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.393955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4c467b-xhthc" event={"ID":"8cea2fb1-54d4-4140-a019-9c3f36396d08","Type":"ContainerDied","Data":"5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03"}
Sep 16 12:50:15.394152 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.394005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4c467b-xhthc" event={"ID":"8cea2fb1-54d4-4140-a019-9c3f36396d08","Type":"ContainerDied","Data":"3e1fa89ce979fb5bc7961af9a48e16b5de61a7df300598853514a7b25c6749d0"}
Sep 16 12:50:15.394152 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.394026    2577 scope.go:117] "RemoveContainer" containerID="5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03"
Sep 16 12:50:15.402729 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.402714    2577 scope.go:117] "RemoveContainer" containerID="5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03"
Sep 16 12:50:15.402998 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:50:15.402979    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03\": container with ID starting with 5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03 not found: ID does not exist" containerID="5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03"
Sep 16 12:50:15.403050 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.403006    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03"} err="failed to get container status \"5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03\": rpc error: code = NotFound desc = could not find container \"5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03\": container with ID starting with 5091a60e0f20cb68a3e4007e27693c90ef7fdb82422527f2541e75a098636d03 not found: ID does not exist"
Sep 16 12:50:15.414099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.414075    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4c467b-xhthc"]
Sep 16 12:50:15.420434 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:15.420408    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56b4c467b-xhthc"]
Sep 16 12:50:16.574971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:16.574936    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cea2fb1-54d4-4140-a019-9c3f36396d08" path="/var/lib/kubelet/pods/8cea2fb1-54d4-4140-a019-9c3f36396d08/volumes"
Sep 16 12:50:20.302736 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:20.302705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:50:20.306508 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:20.306483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75f78dbf54-4bw5b"
Sep 16 12:50:57.508462 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.508432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:50:57.508852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.508719    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cea2fb1-54d4-4140-a019-9c3f36396d08" containerName="console"
Sep 16 12:50:57.508852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.508730    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cea2fb1-54d4-4140-a019-9c3f36396d08" containerName="console"
Sep 16 12:50:57.508852 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.508783    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cea2fb1-54d4-4140-a019-9c3f36396d08" containerName="console"
Sep 16 12:50:57.513428 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.513408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.516083 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.516061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 12:50:57.516398 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.516374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 12:50:57.516451 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.516415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 12:50:57.516758 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.516737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 12:50:57.517007 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.516991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 12:50:57.517090 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.517010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 12:50:57.517151 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.517098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 12:50:57.517355 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.517340    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jk8nq\""
Sep 16 12:50:57.522640 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.522617    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 12:50:57.523248 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.523234    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 12:50:57.532698 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.532680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:50:57.586137 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfnh\" (UniqueName: \"kubernetes.io/projected/ba3b4bde-d865-4638-8e83-21fd26085131-kube-api-access-8rfnh\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-web-config\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba3b4bde-d865-4638-8e83-21fd26085131-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586273 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba3b4bde-d865-4638-8e83-21fd26085131-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba3b4bde-d865-4638-8e83-21fd26085131-config-out\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3b4bde-d865-4638-8e83-21fd26085131-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba3b4bde-d865-4638-8e83-21fd26085131-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.586431 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.586422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687166 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3b4bde-d865-4638-8e83-21fd26085131-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687277 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba3b4bde-d865-4638-8e83-21fd26085131-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687277 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687277 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687277 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687277 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfnh\" (UniqueName: \"kubernetes.io/projected/ba3b4bde-d865-4638-8e83-21fd26085131-kube-api-access-8rfnh\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-web-config\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba3b4bde-d865-4638-8e83-21fd26085131-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba3b4bde-d865-4638-8e83-21fd26085131-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687495 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba3b4bde-d865-4638-8e83-21fd26085131-config-out\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.687783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.687495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.688360 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.688332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba3b4bde-d865-4638-8e83-21fd26085131-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.689161 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.689138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba3b4bde-d865-4638-8e83-21fd26085131-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.689605 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.689584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba3b4bde-d865-4638-8e83-21fd26085131-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.690832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.690832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.690832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba3b4bde-d865-4638-8e83-21fd26085131-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.690832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.690832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-web-config\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.690832 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.691111 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.690897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.692631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.692613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba3b4bde-d865-4638-8e83-21fd26085131-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.692631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.692621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba3b4bde-d865-4638-8e83-21fd26085131-config-out\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.698758 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.698743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfnh\" (UniqueName: \"kubernetes.io/projected/ba3b4bde-d865-4638-8e83-21fd26085131-kube-api-access-8rfnh\") pod \"alertmanager-main-0\" (UID: \"ba3b4bde-d865-4638-8e83-21fd26085131\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:57.824364 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:57.824303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:50:58.102024 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:58.101997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:50:58.104580 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:50:58.104551    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3b4bde_d865_4638_8e83_21fd26085131.slice/crio-b7ba53efae878f4cc8b28c7d81cafce822cba6e091a2720831ca966191b3bce4 WatchSource:0}: Error finding container b7ba53efae878f4cc8b28c7d81cafce822cba6e091a2720831ca966191b3bce4: Status 404 returned error can't find the container with id b7ba53efae878f4cc8b28c7d81cafce822cba6e091a2720831ca966191b3bce4
Sep 16 12:50:58.560242 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:58.560206    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba3b4bde-d865-4638-8e83-21fd26085131" containerID="451a17c9a8ff74424e3c4ca385d5d082730554c4776e49e372f9eb2adb11c362" exitCode=0
Sep 16 12:50:58.560577 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:58.560275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerDied","Data":"451a17c9a8ff74424e3c4ca385d5d082730554c4776e49e372f9eb2adb11c362"}
Sep 16 12:50:58.560577 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:50:58.560339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"b7ba53efae878f4cc8b28c7d81cafce822cba6e091a2720831ca966191b3bce4"}
Sep 16 12:51:00.578046 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.577960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"f0c49d95f21724aa3edcc61ca1abf94b737c3ddf48ef00e5309514e54748b946"}
Sep 16 12:51:00.578046 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.578039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"5eabffef342a0ab4df2641c444f66a301911066dfd55a122c4e008612b75072d"}
Sep 16 12:51:00.578433 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.578080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"3b849707baa1c192b1f65443b0d735ac830f634154b5443c74561471b527ef3b"}
Sep 16 12:51:00.578433 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.578117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"4fcbf647115731f72b7146d16a14d5fcfef0e36e51d4cc9d70012199088dd0b1"}
Sep 16 12:51:00.578433 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.578152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"ff3933909a2d2736ff75a907f8ceb250f666a3cfcfb63f7160acf9d629ff23af"}
Sep 16 12:51:00.578433 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.578188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba3b4bde-d865-4638-8e83-21fd26085131","Type":"ContainerStarted","Data":"ca1e43d3eef6dfc0995c649aecaf87b59e59b32a2efad112688d7396f67c4a35"}
Sep 16 12:51:00.606508 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:00.606462    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.178593648 podStartE2EDuration="3.606447471s" totalImagesPullingTime="1.413862984s" totalInitContainerRuntime="13.990839ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:00.604477225 +0000 UTC m=+188.554822779" watchObservedRunningTime="2026-09-16 12:51:00.606447471 +0000 UTC m=+188.556793063"
Sep 16 12:51:02.302868 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.302834    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86944789d4-hw66k"]
Sep 16 12:51:02.305199 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.305175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.307685 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.307663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d5xsr\""
Sep 16 12:51:02.307964 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.307944    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:51:02.308845 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.308825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:51:02.310021 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.309999    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:51:02.310479 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.310283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:51:02.310570 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.310294    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:51:02.313777 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.313753    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:51:02.318965 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.318946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86944789d4-hw66k"]
Sep 16 12:51:02.432920 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.432896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-config\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.433035 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.432924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbs6\" (UniqueName: \"kubernetes.io/projected/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-kube-api-access-8vbs6\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.433035 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.432942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-oauth-config\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.433035 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.432970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-service-ca\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.433035 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.433016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-trusted-ca-bundle\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.433163 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.433092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-oauth-serving-cert\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.433163 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.433126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-serving-cert\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534196 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-service-ca\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534313 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-trusted-ca-bundle\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534313 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-oauth-serving-cert\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534313 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-serving-cert\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534448 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-config\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534448 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbs6\" (UniqueName: \"kubernetes.io/projected/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-kube-api-access-8vbs6\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.534448 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.534377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-oauth-config\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.535054 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.535024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-service-ca\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.535160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.535050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-oauth-serving-cert\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.535160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.535148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-trusted-ca-bundle\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.535232 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.535159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-config\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.536843 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.536817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-serving-cert\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.536943 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.536895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-oauth-config\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.546753 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.546730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbs6\" (UniqueName: \"kubernetes.io/projected/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-kube-api-access-8vbs6\") pod \"console-86944789d4-hw66k\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") " pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.615950 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.615899    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:02.890043 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:02.889972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86944789d4-hw66k"]
Sep 16 12:51:02.893057 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:51:02.893026    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b84be8_c13e_49d1_83dd_5a3af10b2a7f.slice/crio-39c9ebff0ae816da1fb9ffd6e281700b81f9f0891896654b13f0e1ba9899e218 WatchSource:0}: Error finding container 39c9ebff0ae816da1fb9ffd6e281700b81f9f0891896654b13f0e1ba9899e218: Status 404 returned error can't find the container with id 39c9ebff0ae816da1fb9ffd6e281700b81f9f0891896654b13f0e1ba9899e218
Sep 16 12:51:03.591576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:03.591547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86944789d4-hw66k" event={"ID":"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f","Type":"ContainerStarted","Data":"be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318"}
Sep 16 12:51:03.591980 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:03.591591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86944789d4-hw66k" event={"ID":"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f","Type":"ContainerStarted","Data":"39c9ebff0ae816da1fb9ffd6e281700b81f9f0891896654b13f0e1ba9899e218"}
Sep 16 12:51:03.612695 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:03.612650    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86944789d4-hw66k" podStartSLOduration=1.612635981 podStartE2EDuration="1.612635981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:03.61047402 +0000 UTC m=+191.560819573" watchObservedRunningTime="2026-09-16 12:51:03.612635981 +0000 UTC m=+191.562981533"
Sep 16 12:51:12.616758 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:12.616726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:12.617153 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:12.616786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:12.621348 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:12.621326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:51:12.628943 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:51:12.628925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:52:52.468319 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:52:52.468289    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:52:52.472071 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:52:52.472045    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:52:52.479141 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:52:52.479122    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:53:15.826955 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.826921    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"]
Sep 16 12:53:15.830766 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.830743    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:15.834378 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.834349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jg5wq\""
Sep 16 12:53:15.834499 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.834477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 12:53:15.834818 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.834805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 12:53:15.835438 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.835424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 12:53:15.850326 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:15.850302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"]
Sep 16 12:53:16.004289 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.004261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzw8\" (UniqueName: \"kubernetes.io/projected/d9db7ffa-a0f8-48d7-a929-0d074eacb3e6-kube-api-access-7xzw8\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2s7mk\" (UID: \"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.004421 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.004307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d9db7ffa-a0f8-48d7-a929-0d074eacb3e6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2s7mk\" (UID: \"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.104989 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.104910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d9db7ffa-a0f8-48d7-a929-0d074eacb3e6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2s7mk\" (UID: \"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.105136 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.104986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzw8\" (UniqueName: \"kubernetes.io/projected/d9db7ffa-a0f8-48d7-a929-0d074eacb3e6-kube-api-access-7xzw8\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2s7mk\" (UID: \"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.107443 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.107417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d9db7ffa-a0f8-48d7-a929-0d074eacb3e6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2s7mk\" (UID: \"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.133808 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.133787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzw8\" (UniqueName: \"kubernetes.io/projected/d9db7ffa-a0f8-48d7-a929-0d074eacb3e6-kube-api-access-7xzw8\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-2s7mk\" (UID: \"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.140626 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.140608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:16.439110 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.439065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"]
Sep 16 12:53:16.441391 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:53:16.441365    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9db7ffa_a0f8_48d7_a929_0d074eacb3e6.slice/crio-a5e8c2e6cd9d20dac41174d371655272ad899fef9a3618417fb8e1a03b48fa50 WatchSource:0}: Error finding container a5e8c2e6cd9d20dac41174d371655272ad899fef9a3618417fb8e1a03b48fa50: Status 404 returned error can't find the container with id a5e8c2e6cd9d20dac41174d371655272ad899fef9a3618417fb8e1a03b48fa50
Sep 16 12:53:16.443192 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:16.443169    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:53:17.117229 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:17.117192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk" event={"ID":"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6","Type":"ContainerStarted","Data":"a5e8c2e6cd9d20dac41174d371655272ad899fef9a3618417fb8e1a03b48fa50"}
Sep 16 12:53:20.141054 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.141021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk" event={"ID":"d9db7ffa-a0f8-48d7-a929-0d074eacb3e6","Type":"ContainerStarted","Data":"e5c3ce09a18eae6c7a1f8f4c64ea36f102322314b9ad86724110c5a9568ffae5"}
Sep 16 12:53:20.141517 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.141130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:53:20.176978 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.176922    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk" podStartSLOduration=1.9517972270000001 podStartE2EDuration="5.176908918s" totalImagesPullingTime="3.225111691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:53:20.175710781 +0000 UTC m=+328.126056333" watchObservedRunningTime="2026-09-16 12:53:20.176908918 +0000 UTC m=+328.127254486"
Sep 16 12:53:20.628731 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.628699    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"]
Sep 16 12:53:20.632108 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.632085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.636924 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.636903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-gjmzf\""
Sep 16 12:53:20.637220 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.637204    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 12:53:20.637322 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.637219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 12:53:20.646872 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.646835    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"]
Sep 16 12:53:20.647040 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.646910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvld5\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-kube-api-access-gvld5\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.647177 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.647159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.647312 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.647295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4b582df5-887a-4f41-b749-5b95e655a9a4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.747871 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.747822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvld5\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-kube-api-access-gvld5\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.747871 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.747874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.748096 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.747935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4b582df5-887a-4f41-b749-5b95e655a9a4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.748096 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:20.748053    2577 secret.go:281] references non-existent secret key: tls.crt
Sep 16 12:53:20.748096 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:20.748074    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 12:53:20.748096 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:20.748093    2577 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 16 12:53:20.748283 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:20.748116    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 16 12:53:20.748283 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:20.748216    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates podName:4b582df5-887a-4f41-b749-5b95e655a9a4 nodeName:}" failed. No retries permitted until 2026-09-16 12:53:21.248188532 +0000 UTC m=+329.198534082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates") pod "keda-metrics-apiserver-77857b7b98-65gpx" (UID: "4b582df5-887a-4f41-b749-5b95e655a9a4") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 16 12:53:20.748394 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.748348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4b582df5-887a-4f41-b749-5b95e655a9a4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:20.757066 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:20.757035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvld5\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-kube-api-access-gvld5\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:21.252392 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:21.252357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:21.252785 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:21.252521    2577 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 16 12:53:21.252785 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:21.252549    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx: secret "keda-metrics-apiserver-certs" not found
Sep 16 12:53:21.252785 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:53:21.252613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates podName:4b582df5-887a-4f41-b749-5b95e655a9a4 nodeName:}" failed. No retries permitted until 2026-09-16 12:53:22.252592544 +0000 UTC m=+330.202938081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates") pod "keda-metrics-apiserver-77857b7b98-65gpx" (UID: "4b582df5-887a-4f41-b749-5b95e655a9a4") : secret "keda-metrics-apiserver-certs" not found
Sep 16 12:53:22.260399 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:22.260360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:22.262921 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:22.262894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b582df5-887a-4f41-b749-5b95e655a9a4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-65gpx\" (UID: \"4b582df5-887a-4f41-b749-5b95e655a9a4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:22.442892 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:22.442835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:22.757005 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:22.756944    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"]
Sep 16 12:53:22.759160 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:53:22.759136    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b582df5_887a_4f41_b749_5b95e655a9a4.slice/crio-5f1029834887069a01bcf1244ae33297ff19a655137c19ace3c4fb4c55ad654b WatchSource:0}: Error finding container 5f1029834887069a01bcf1244ae33297ff19a655137c19ace3c4fb4c55ad654b: Status 404 returned error can't find the container with id 5f1029834887069a01bcf1244ae33297ff19a655137c19ace3c4fb4c55ad654b
Sep 16 12:53:23.155196 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:23.155112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx" event={"ID":"4b582df5-887a-4f41-b749-5b95e655a9a4","Type":"ContainerStarted","Data":"5f1029834887069a01bcf1244ae33297ff19a655137c19ace3c4fb4c55ad654b"}
Sep 16 12:53:26.173620 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:26.173579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx" event={"ID":"4b582df5-887a-4f41-b749-5b95e655a9a4","Type":"ContainerStarted","Data":"896aa91a14092802fd9564a15ff891ef8ada06e5a80e539a15a2945842ffafeb"}
Sep 16 12:53:26.174090 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:26.173714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:26.194157 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:26.194105    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx" podStartSLOduration=3.557908591 podStartE2EDuration="6.194086866s" totalImagesPullingTime="2.636178275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:53:26.192316487 +0000 UTC m=+334.142662030" watchObservedRunningTime="2026-09-16 12:53:26.194086866 +0000 UTC m=+334.144432419"
Sep 16 12:53:37.183369 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:37.183339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-65gpx"
Sep 16 12:53:41.147233 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:53:41.147198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-2s7mk"
Sep 16 12:54:21.504569 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.504535    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595b95f48b-5tqfb"]
Sep 16 12:54:21.508179 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.508160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.540471 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.540438    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595b95f48b-5tqfb"]
Sep 16 12:54:21.565232 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-service-ca\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.565359 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txzs\" (UniqueName: \"kubernetes.io/projected/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-kube-api-access-6txzs\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.565359 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-oauth-config\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.565471 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-trusted-ca-bundle\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.565471 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-oauth-serving-cert\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.565575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-serving-cert\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.565575 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.565522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-config\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.666971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.666942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-oauth-serving-cert\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.666971 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.666976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-serving-cert\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.667176 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.666992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-config\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.667176 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.667029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-service-ca\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.667176 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.667059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txzs\" (UniqueName: \"kubernetes.io/projected/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-kube-api-access-6txzs\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.667176 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.667143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-oauth-config\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.667176 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.667176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-trusted-ca-bundle\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.667963 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.667935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-config\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.668074 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.667973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-oauth-serving-cert\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.668074 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.668032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-service-ca\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.668362 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.668084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-trusted-ca-bundle\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.669564 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.669544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-serving-cert\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.669834 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.669817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-console-oauth-config\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.730388 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.730367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txzs\" (UniqueName: \"kubernetes.io/projected/fb47497e-18ab-43f1-bc6c-f098ea1c46a5-kube-api-access-6txzs\") pod \"console-595b95f48b-5tqfb\" (UID: \"fb47497e-18ab-43f1-bc6c-f098ea1c46a5\") " pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:21.816900 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:21.816796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:22.279656 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:22.279630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595b95f48b-5tqfb"]
Sep 16 12:54:22.280296 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:54:22.280249    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb47497e_18ab_43f1_bc6c_f098ea1c46a5.slice/crio-414fb64ab1bc95594f5b422f99d272275bfb5dac25cae05d94faa00d50577e99 WatchSource:0}: Error finding container 414fb64ab1bc95594f5b422f99d272275bfb5dac25cae05d94faa00d50577e99: Status 404 returned error can't find the container with id 414fb64ab1bc95594f5b422f99d272275bfb5dac25cae05d94faa00d50577e99
Sep 16 12:54:22.438350 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:22.438311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595b95f48b-5tqfb" event={"ID":"fb47497e-18ab-43f1-bc6c-f098ea1c46a5","Type":"ContainerStarted","Data":"fb9e5bb2c184a376493496007d14d481a29720789ac8581bd2a57492155e8ffc"}
Sep 16 12:54:22.438501 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:22.438369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595b95f48b-5tqfb" event={"ID":"fb47497e-18ab-43f1-bc6c-f098ea1c46a5","Type":"ContainerStarted","Data":"414fb64ab1bc95594f5b422f99d272275bfb5dac25cae05d94faa00d50577e99"}
Sep 16 12:54:22.498936 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:22.498879    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595b95f48b-5tqfb" podStartSLOduration=1.498843661 podStartE2EDuration="1.498843661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:22.498009878 +0000 UTC m=+390.448355430" watchObservedRunningTime="2026-09-16 12:54:22.498843661 +0000 UTC m=+390.449189214"
Sep 16 12:54:31.817085 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:31.817051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:31.817557 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:31.817128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:31.821968 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:31.821945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:32.484392 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:32.484363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595b95f48b-5tqfb"
Sep 16 12:54:32.772614 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:32.772538    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86944789d4-hw66k"]
Sep 16 12:54:54.202532 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.202496    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867644d759-r5752"]
Sep 16 12:54:54.206121 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.206098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.209681 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.209654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 12:54:54.209918 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.209894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-bp5wr\""
Sep 16 12:54:54.210569 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.210550    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 12:54:54.210897 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.210852    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 12:54:54.221052 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.221029    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-r5752"]
Sep 16 12:54:54.342404 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.342364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3fef098-a412-43a3-aae7-41c9c360f8e8-cert\") pod \"kserve-controller-manager-867644d759-r5752\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") " pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.342588 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.342421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9gsj\" (UniqueName: \"kubernetes.io/projected/e3fef098-a412-43a3-aae7-41c9c360f8e8-kube-api-access-h9gsj\") pod \"kserve-controller-manager-867644d759-r5752\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") " pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.443965 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.443935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3fef098-a412-43a3-aae7-41c9c360f8e8-cert\") pod \"kserve-controller-manager-867644d759-r5752\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") " pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.444104 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.443976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9gsj\" (UniqueName: \"kubernetes.io/projected/e3fef098-a412-43a3-aae7-41c9c360f8e8-kube-api-access-h9gsj\") pod \"kserve-controller-manager-867644d759-r5752\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") " pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.446556 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.446533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3fef098-a412-43a3-aae7-41c9c360f8e8-cert\") pod \"kserve-controller-manager-867644d759-r5752\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") " pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.454488 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.454432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9gsj\" (UniqueName: \"kubernetes.io/projected/e3fef098-a412-43a3-aae7-41c9c360f8e8-kube-api-access-h9gsj\") pod \"kserve-controller-manager-867644d759-r5752\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") " pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.518110 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.518086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:54.794891 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:54.794793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-r5752"]
Sep 16 12:54:54.801639 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:54:54.801614    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fef098_a412_43a3_aae7_41c9c360f8e8.slice/crio-dfed621fee7296e89fa66cf952f773cb8e7ea6cb8710c04e0592d90e4c0ec9e2 WatchSource:0}: Error finding container dfed621fee7296e89fa66cf952f773cb8e7ea6cb8710c04e0592d90e4c0ec9e2: Status 404 returned error can't find the container with id dfed621fee7296e89fa66cf952f773cb8e7ea6cb8710c04e0592d90e4c0ec9e2
Sep 16 12:54:55.591327 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:55.591286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-r5752" event={"ID":"e3fef098-a412-43a3-aae7-41c9c360f8e8","Type":"ContainerStarted","Data":"dfed621fee7296e89fa66cf952f773cb8e7ea6cb8710c04e0592d90e4c0ec9e2"}
Sep 16 12:54:57.603133 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:57.603100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-r5752" event={"ID":"e3fef098-a412-43a3-aae7-41c9c360f8e8","Type":"ContainerStarted","Data":"d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18"}
Sep 16 12:54:57.603576 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:57.603184    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:54:57.621990 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:57.621940    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867644d759-r5752" podStartSLOduration=0.980766546 podStartE2EDuration="3.621922041s" totalImagesPullingTime="2.641155495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:57.621697358 +0000 UTC m=+425.572042912" watchObservedRunningTime="2026-09-16 12:54:57.621922041 +0000 UTC m=+425.572267596"
Sep 16 12:54:57.791427 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:57.791351    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86944789d4-hw66k" podUID="b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" containerName="console" containerID="cri-o://be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318" gracePeriod=15
Sep 16 12:54:58.027826 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.027803    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:54:58.181151 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-oauth-serving-cert\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181151 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vbs6\" (UniqueName: \"kubernetes.io/projected/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-kube-api-access-8vbs6\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181151 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-config\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181415 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-serving-cert\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181415 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-oauth-config\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181415 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-service-ca\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181415 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181279    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-trusted-ca-bundle\") pod \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\" (UID: \"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f\") "
Sep 16 12:54:58.181631 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:54:58.181683 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-config" (OuterVolumeSpecName: "console-config") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:54:58.181683 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:54:58.182103 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.181964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:54:58.183519 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.183499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:54:58.183601 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.183548    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:54:58.183601 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.183562    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-kube-api-access-8vbs6" (OuterVolumeSpecName: "kube-api-access-8vbs6") pod "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" (UID: "b4b84be8-c13e-49d1-83dd-5a3af10b2a7f"). InnerVolumeSpecName "kube-api-access-8vbs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:54:58.282133 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282110    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-service-ca\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.282133 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282132    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-trusted-ca-bundle\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.282286 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282141    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-oauth-serving-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.282286 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282149    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbs6\" (UniqueName: \"kubernetes.io/projected/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-kube-api-access-8vbs6\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.282286 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282158    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.282286 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282166    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-serving-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.282286 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.282173    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f-console-oauth-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:54:58.610726 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.610696    2577 generic.go:356] "Generic (PLEG): container finished" podID="b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" containerID="be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318" exitCode=0
Sep 16 12:54:58.611099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.610749    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86944789d4-hw66k"
Sep 16 12:54:58.611099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.610777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86944789d4-hw66k" event={"ID":"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f","Type":"ContainerDied","Data":"be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318"}
Sep 16 12:54:58.611099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.610844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86944789d4-hw66k" event={"ID":"b4b84be8-c13e-49d1-83dd-5a3af10b2a7f","Type":"ContainerDied","Data":"39c9ebff0ae816da1fb9ffd6e281700b81f9f0891896654b13f0e1ba9899e218"}
Sep 16 12:54:58.611099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.610882    2577 scope.go:117] "RemoveContainer" containerID="be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318"
Sep 16 12:54:58.619078 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.619065    2577 scope.go:117] "RemoveContainer" containerID="be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318"
Sep 16 12:54:58.619328 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:54:58.619312    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318\": container with ID starting with be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318 not found: ID does not exist" containerID="be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318"
Sep 16 12:54:58.619393 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.619334    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318"} err="failed to get container status \"be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318\": rpc error: code = NotFound desc = could not find container \"be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318\": container with ID starting with be5e9c37f51ce7ba4e5da4f8d81987236e7c5ebd20232faaeed6666cbd9c7318 not found: ID does not exist"
Sep 16 12:54:58.628099 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.628080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86944789d4-hw66k"]
Sep 16 12:54:58.631936 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:54:58.631914    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86944789d4-hw66k"]
Sep 16 12:55:00.573551 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:00.573519    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" path="/var/lib/kubelet/pods/b4b84be8-c13e-49d1-83dd-5a3af10b2a7f/volumes"
Sep 16 12:55:28.616614 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:28.616532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:55:40.118358 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.118328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867644d759-r5752"]
Sep 16 12:55:40.118791 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.118542    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-867644d759-r5752" podUID="e3fef098-a412-43a3-aae7-41c9c360f8e8" containerName="manager" containerID="cri-o://d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18" gracePeriod=10
Sep 16 12:55:40.148108 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.148080    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867644d759-qdzjt"]
Sep 16 12:55:40.148472 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.148455    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" containerName="console"
Sep 16 12:55:40.148548 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.148474    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" containerName="console"
Sep 16 12:55:40.148604 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.148597    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b84be8-c13e-49d1-83dd-5a3af10b2a7f" containerName="console"
Sep 16 12:55:40.151400 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.151381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.165156 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.165123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-qdzjt"]
Sep 16 12:55:40.221963 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.221931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad550311-c5c2-4173-8ae2-4875cbdadd1d-cert\") pod \"kserve-controller-manager-867644d759-qdzjt\" (UID: \"ad550311-c5c2-4173-8ae2-4875cbdadd1d\") " pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.222055 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.221978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4bq\" (UniqueName: \"kubernetes.io/projected/ad550311-c5c2-4173-8ae2-4875cbdadd1d-kube-api-access-br4bq\") pod \"kserve-controller-manager-867644d759-qdzjt\" (UID: \"ad550311-c5c2-4173-8ae2-4875cbdadd1d\") " pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.323428 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.323392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad550311-c5c2-4173-8ae2-4875cbdadd1d-cert\") pod \"kserve-controller-manager-867644d759-qdzjt\" (UID: \"ad550311-c5c2-4173-8ae2-4875cbdadd1d\") " pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.323589 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.323452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4bq\" (UniqueName: \"kubernetes.io/projected/ad550311-c5c2-4173-8ae2-4875cbdadd1d-kube-api-access-br4bq\") pod \"kserve-controller-manager-867644d759-qdzjt\" (UID: \"ad550311-c5c2-4173-8ae2-4875cbdadd1d\") " pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.325948 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.325918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad550311-c5c2-4173-8ae2-4875cbdadd1d-cert\") pod \"kserve-controller-manager-867644d759-qdzjt\" (UID: \"ad550311-c5c2-4173-8ae2-4875cbdadd1d\") " pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.332705 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.332680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4bq\" (UniqueName: \"kubernetes.io/projected/ad550311-c5c2-4173-8ae2-4875cbdadd1d-kube-api-access-br4bq\") pod \"kserve-controller-manager-867644d759-qdzjt\" (UID: \"ad550311-c5c2-4173-8ae2-4875cbdadd1d\") " pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.353491 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.353474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:55:40.424169 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.424098    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3fef098-a412-43a3-aae7-41c9c360f8e8-cert\") pod \"e3fef098-a412-43a3-aae7-41c9c360f8e8\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") "
Sep 16 12:55:40.424169 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.424138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9gsj\" (UniqueName: \"kubernetes.io/projected/e3fef098-a412-43a3-aae7-41c9c360f8e8-kube-api-access-h9gsj\") pod \"e3fef098-a412-43a3-aae7-41c9c360f8e8\" (UID: \"e3fef098-a412-43a3-aae7-41c9c360f8e8\") "
Sep 16 12:55:40.426309 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.426285    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fef098-a412-43a3-aae7-41c9c360f8e8-kube-api-access-h9gsj" (OuterVolumeSpecName: "kube-api-access-h9gsj") pod "e3fef098-a412-43a3-aae7-41c9c360f8e8" (UID: "e3fef098-a412-43a3-aae7-41c9c360f8e8"). InnerVolumeSpecName "kube-api-access-h9gsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:40.426394 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.426316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fef098-a412-43a3-aae7-41c9c360f8e8-cert" (OuterVolumeSpecName: "cert") pod "e3fef098-a412-43a3-aae7-41c9c360f8e8" (UID: "e3fef098-a412-43a3-aae7-41c9c360f8e8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:40.491424 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.491396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:40.525697 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.525669    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3fef098-a412-43a3-aae7-41c9c360f8e8-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:55:40.525804 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.525703    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9gsj\" (UniqueName: \"kubernetes.io/projected/e3fef098-a412-43a3-aae7-41c9c360f8e8-kube-api-access-h9gsj\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 12:55:40.767610 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.767583    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-qdzjt"]
Sep 16 12:55:40.769358 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:55:40.769336    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad550311_c5c2_4173_8ae2_4875cbdadd1d.slice/crio-f54853a6863b07306f356640e49251391cc20a78c4733b1b8b126e8c75398682 WatchSource:0}: Error finding container f54853a6863b07306f356640e49251391cc20a78c4733b1b8b126e8c75398682: Status 404 returned error can't find the container with id f54853a6863b07306f356640e49251391cc20a78c4733b1b8b126e8c75398682
Sep 16 12:55:40.792849 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.792812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-qdzjt" event={"ID":"ad550311-c5c2-4173-8ae2-4875cbdadd1d","Type":"ContainerStarted","Data":"f54853a6863b07306f356640e49251391cc20a78c4733b1b8b126e8c75398682"}
Sep 16 12:55:40.794070 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.794047    2577 generic.go:356] "Generic (PLEG): container finished" podID="e3fef098-a412-43a3-aae7-41c9c360f8e8" containerID="d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18" exitCode=0
Sep 16 12:55:40.794145 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.794092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-r5752"
Sep 16 12:55:40.794183 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.794095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-r5752" event={"ID":"e3fef098-a412-43a3-aae7-41c9c360f8e8","Type":"ContainerDied","Data":"d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18"}
Sep 16 12:55:40.794221 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.794210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-r5752" event={"ID":"e3fef098-a412-43a3-aae7-41c9c360f8e8","Type":"ContainerDied","Data":"dfed621fee7296e89fa66cf952f773cb8e7ea6cb8710c04e0592d90e4c0ec9e2"}
Sep 16 12:55:40.794254 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.794234    2577 scope.go:117] "RemoveContainer" containerID="d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18"
Sep 16 12:55:40.802778 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.802759    2577 scope.go:117] "RemoveContainer" containerID="d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18"
Sep 16 12:55:40.803090 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:55:40.803062    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18\": container with ID starting with d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18 not found: ID does not exist" containerID="d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18"
Sep 16 12:55:40.803156 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.803091    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18"} err="failed to get container status \"d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18\": rpc error: code = NotFound desc = could not find container \"d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18\": container with ID starting with d7b671937a4781fd8af63d1b05a07800042680953ee3e051248d8bd9bbdaaf18 not found: ID does not exist"
Sep 16 12:55:40.815417 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.815386    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867644d759-r5752"]
Sep 16 12:55:40.825027 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:40.825007    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-867644d759-r5752"]
Sep 16 12:55:41.800329 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:41.800292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-qdzjt" event={"ID":"ad550311-c5c2-4173-8ae2-4875cbdadd1d","Type":"ContainerStarted","Data":"bcaa55e900ca21c27d7145c5ce6bbbc8a26682f108939f5ffcbc97d1b841002a"}
Sep 16 12:55:41.800790 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:41.800454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:55:41.819123 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:41.819069    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867644d759-qdzjt" podStartSLOduration=1.436290818 podStartE2EDuration="1.819051828s" totalImagesPullingTime="382.76101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:41.818274256 +0000 UTC m=+469.768619810" watchObservedRunningTime="2026-09-16 12:55:41.819051828 +0000 UTC m=+469.769397375"
Sep 16 12:55:42.572395 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:55:42.572354    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3fef098-a412-43a3-aae7-41c9c360f8e8" path="/var/lib/kubelet/pods/e3fef098-a412-43a3-aae7-41c9c360f8e8/volumes"
Sep 16 12:56:12.811841 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:12.811813    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867644d759-qdzjt"
Sep 16 12:56:14.232232 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.232203    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-v94xv"]
Sep 16 12:56:14.232687 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.232572    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3fef098-a412-43a3-aae7-41c9c360f8e8" containerName="manager"
Sep 16 12:56:14.232687 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.232584    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3fef098-a412-43a3-aae7-41c9c360f8e8" containerName="manager"
Sep 16 12:56:14.232687 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.232653    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3fef098-a412-43a3-aae7-41c9c360f8e8" containerName="manager"
Sep 16 12:56:14.236806 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.236789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.240567 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.240543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 12:56:14.241068 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.241052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-h2rn2\""
Sep 16 12:56:14.250811 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.250786    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-v94xv"]
Sep 16 12:56:14.320441 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.320406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056515a3-501a-4f54-b8c9-2b5a3b166396-tls-certs\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.320556 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.320489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs4tm\" (UniqueName: \"kubernetes.io/projected/056515a3-501a-4f54-b8c9-2b5a3b166396-kube-api-access-rs4tm\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.421562 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.421538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056515a3-501a-4f54-b8c9-2b5a3b166396-tls-certs\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.421697 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.421577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs4tm\" (UniqueName: \"kubernetes.io/projected/056515a3-501a-4f54-b8c9-2b5a3b166396-kube-api-access-rs4tm\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.421741 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:56:14.421695    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 12:56:14.421781 ip-10-0-143-99 kubenswrapper[2577]: E0916 12:56:14.421769    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/056515a3-501a-4f54-b8c9-2b5a3b166396-tls-certs podName:056515a3-501a-4f54-b8c9-2b5a3b166396 nodeName:}" failed. No retries permitted until 2026-09-16 12:56:14.921750385 +0000 UTC m=+502.872095916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/056515a3-501a-4f54-b8c9-2b5a3b166396-tls-certs") pod "model-serving-api-7b954ffc95-v94xv" (UID: "056515a3-501a-4f54-b8c9-2b5a3b166396") : secret "model-serving-api-tls" not found
Sep 16 12:56:14.439043 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.439020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs4tm\" (UniqueName: \"kubernetes.io/projected/056515a3-501a-4f54-b8c9-2b5a3b166396-kube-api-access-rs4tm\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.925022 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.924989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056515a3-501a-4f54-b8c9-2b5a3b166396-tls-certs\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:14.927426 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:14.927397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056515a3-501a-4f54-b8c9-2b5a3b166396-tls-certs\") pod \"model-serving-api-7b954ffc95-v94xv\" (UID: \"056515a3-501a-4f54-b8c9-2b5a3b166396\") " pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:15.148207 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:15.148178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:15.425911 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:15.425886    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-v94xv"]
Sep 16 12:56:15.427586 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:56:15.427560    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056515a3_501a_4f54_b8c9_2b5a3b166396.slice/crio-5acbd37d4c94760e125000f9128f545048df46733128c72caba782d4d5debb06 WatchSource:0}: Error finding container 5acbd37d4c94760e125000f9128f545048df46733128c72caba782d4d5debb06: Status 404 returned error can't find the container with id 5acbd37d4c94760e125000f9128f545048df46733128c72caba782d4d5debb06
Sep 16 12:56:15.954501 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:15.954467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-v94xv" event={"ID":"056515a3-501a-4f54-b8c9-2b5a3b166396","Type":"ContainerStarted","Data":"5acbd37d4c94760e125000f9128f545048df46733128c72caba782d4d5debb06"}
Sep 16 12:56:17.967167 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:17.967132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-v94xv" event={"ID":"056515a3-501a-4f54-b8c9-2b5a3b166396","Type":"ContainerStarted","Data":"5dfb5e2cf3b08c300179a6c3de8db6a8d3f3b18fba33c5b7ade8445e9a0a92bb"}
Sep 16 12:56:17.967527 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:17.967267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:56:17.990409 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:17.990365    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-v94xv" podStartSLOduration=1.636407879 podStartE2EDuration="3.990350617s" totalImagesPullingTime="2.353942738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:17.988904259 +0000 UTC m=+505.939249813" watchObservedRunningTime="2026-09-16 12:56:17.990350617 +0000 UTC m=+505.940696169"
Sep 16 12:56:28.976272 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:56:28.976246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-v94xv"
Sep 16 12:57:52.500925 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:57:52.500896    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:57:52.502127 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:57:52.502105    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 12:58:01.131608 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.131571    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"]
Sep 16 12:58:01.142580 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.142553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.146104 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.146081    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 12:58:01.146809 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.146700    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 12:58:01.147069 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.146756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 12:58:01.147490 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.147470    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 12:58:01.147652 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.147638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 12:58:01.147894 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.147850    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9sk5b\""
Sep 16 12:58:01.148142 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.147947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 12:58:01.150233 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.150210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"]
Sep 16 12:58:01.192868 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.192835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwqh\" (UniqueName: \"kubernetes.io/projected/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-api-access-vnwqh\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.193024 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.192904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d94ade4-7161-43cf-95dd-90c92da670b0-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.193024 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.192956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d94ade4-7161-43cf-95dd-90c92da670b0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.193119 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.193040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.193156 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.193122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.294445 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.294414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d94ade4-7161-43cf-95dd-90c92da670b0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.294445 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.294447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.294694 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.294481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.294694 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.294541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwqh\" (UniqueName: \"kubernetes.io/projected/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-api-access-vnwqh\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.294694 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.294580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d94ade4-7161-43cf-95dd-90c92da670b0-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.294953 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.294932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d94ade4-7161-43cf-95dd-90c92da670b0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.295225 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.295202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.295324 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.295263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.297160 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.297139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d94ade4-7161-43cf-95dd-90c92da670b0-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.307031 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.307013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwqh\" (UniqueName: \"kubernetes.io/projected/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-api-access-vnwqh\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-zhm5z\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.457995 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.457972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:01.733430 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:01.733401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"]
Sep 16 12:58:01.735433 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:58:01.735406    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d94ade4_7161_43cf_95dd_90c92da670b0.slice/crio-19be7a9c956c8ec3fe6a10ecb934db4066ebbe22809d923087b22f1c55359c80 WatchSource:0}: Error finding container 19be7a9c956c8ec3fe6a10ecb934db4066ebbe22809d923087b22f1c55359c80: Status 404 returned error can't find the container with id 19be7a9c956c8ec3fe6a10ecb934db4066ebbe22809d923087b22f1c55359c80
Sep 16 12:58:02.427935 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:02.427897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerStarted","Data":"19be7a9c956c8ec3fe6a10ecb934db4066ebbe22809d923087b22f1c55359c80"}
Sep 16 12:58:05.446501 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:05.446470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerStarted","Data":"4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad"}
Sep 16 12:58:09.467504 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:09.467467    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerID="4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad" exitCode=0
Sep 16 12:58:09.467898 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:09.467541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerDied","Data":"4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad"}
Sep 16 12:58:22.398305 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:22.398284    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:58:22.544799 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:22.544759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerStarted","Data":"b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e"}
Sep 16 12:58:25.564818 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:25.564787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerStarted","Data":"0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3"}
Sep 16 12:58:25.565256 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:25.565073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:25.565256 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:25.565120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:25.566179 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:25.566151    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:58:26.569823 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:26.569780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:58:31.574326 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:31.574299    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:58:31.574775 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:31.574735    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:58:31.600724 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:31.600676    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podStartSLOduration=7.95465083 podStartE2EDuration="30.600662592s" totalImagesPullingTime="18.671093106s" totalInitContainerRuntime="3.974918656s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:25.598968418 +0000 UTC m=+633.549313970" watchObservedRunningTime="2026-09-16 12:58:31.600662592 +0000 UTC m=+639.551008144"
Sep 16 12:58:41.574783 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:41.574740    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:58:51.575210 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:58:51.575171    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:59:01.575607 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:01.575566    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:59:11.574793 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:11.574747    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:59:21.575711 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:21.575676    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:59:31.575336 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:31.575299    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 12:59:41.575520 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:41.575491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 12:59:46.273251 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.273222    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"]
Sep 16 12:59:46.276929 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.276909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.280196 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.280173    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-ec80e-kube-rbac-proxy-sar-config\""
Sep 16 12:59:46.281814 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.281796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-ec80e-predictor-serving-cert\""
Sep 16 12:59:46.297572 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.297548    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"]
Sep 16 12:59:46.408378 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.408341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9413682b-75bf-46ba-a4e8-80329f35bf1f-proxy-tls\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.408378 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.408380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c26x\" (UniqueName: \"kubernetes.io/projected/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-api-access-9c26x\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.408604 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.408405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.408604 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.408529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9413682b-75bf-46ba-a4e8-80329f35bf1f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.408604 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.408565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.509368 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.509335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9413682b-75bf-46ba-a4e8-80329f35bf1f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.509368 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.509365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.509573 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.509441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9413682b-75bf-46ba-a4e8-80329f35bf1f-proxy-tls\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.509573 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.509461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c26x\" (UniqueName: \"kubernetes.io/projected/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-api-access-9c26x\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.509573 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.509482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.509797 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.509777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9413682b-75bf-46ba-a4e8-80329f35bf1f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.510189 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.510134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.510189 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.510150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.512019 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.512002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9413682b-75bf-46ba-a4e8-80329f35bf1f-proxy-tls\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.520321 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.520298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c26x\" (UniqueName: \"kubernetes.io/projected/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-api-access-9c26x\") pod \"isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.587151 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.587095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:46.873401 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.873320    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"]
Sep 16 12:59:46.877815 ip-10-0-143-99 kubenswrapper[2577]: W0916 12:59:46.877791    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9413682b_75bf_46ba_a4e8_80329f35bf1f.slice/crio-b5c8cea5450822086efe660912a8e6e6e6cb86bfb0a8ba5de15b9d021b3389ff WatchSource:0}: Error finding container b5c8cea5450822086efe660912a8e6e6e6cb86bfb0a8ba5de15b9d021b3389ff: Status 404 returned error can't find the container with id b5c8cea5450822086efe660912a8e6e6e6cb86bfb0a8ba5de15b9d021b3389ff
Sep 16 12:59:46.938792 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.938768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerStarted","Data":"bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202"}
Sep 16 12:59:46.938921 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:46.938828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerStarted","Data":"b5c8cea5450822086efe660912a8e6e6e6cb86bfb0a8ba5de15b9d021b3389ff"}
Sep 16 12:59:50.963899 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:50.963847    2577 generic.go:356] "Generic (PLEG): container finished" podID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerID="bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202" exitCode=0
Sep 16 12:59:50.964252 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:50.963925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerDied","Data":"bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202"}
Sep 16 12:59:51.970990 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:51.970962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerStarted","Data":"7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500"}
Sep 16 12:59:51.971399 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:51.971015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerStarted","Data":"061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60"}
Sep 16 12:59:51.971399 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:51.971300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:51.971498 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:51.971457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:51.972748 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:51.972721    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 12:59:51.999357 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:51.999317    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podStartSLOduration=2.097525111 podStartE2EDuration="5.999304107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901778996s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:51.998035071 +0000 UTC m=+719.948380625" watchObservedRunningTime="2026-09-16 12:59:51.999304107 +0000 UTC m=+719.949649656"
Sep 16 12:59:52.975809 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:52.975772    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 12:59:57.980310 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:57.980280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 12:59:57.980781 ip-10-0-143-99 kubenswrapper[2577]: I0916 12:59:57.980751    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:00:07.981126 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:00:07.981090    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:00:17.981269 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:00:17.981232    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:00:27.980917 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:00:27.980872    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:00:37.981485 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:00:37.981446    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:00:47.981004 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:00:47.980964    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:00:57.981497 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:00:57.981456    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:01:07.981616 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:07.981587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 13:01:21.032185 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.032104    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"]
Sep 16 13:01:21.032541 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.032363    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" containerID="cri-o://b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e" gracePeriod=30
Sep 16 13:01:21.032541 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.032412    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kube-rbac-proxy" containerID="cri-o://0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3" gracePeriod=30
Sep 16 13:01:21.386014 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.385924    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerID="0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3" exitCode=2
Sep 16 13:01:21.386014 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.385974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerDied","Data":"0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3"}
Sep 16 13:01:21.570198 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.570156    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 13:01:21.575507 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:21.575480    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 13:01:25.171088 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.171065    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 13:01:25.232670 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.232598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnwqh\" (UniqueName: \"kubernetes.io/projected/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-api-access-vnwqh\") pod \"4d94ade4-7161-43cf-95dd-90c92da670b0\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") "
Sep 16 13:01:25.232670 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.232631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-cabundle-cert\") pod \"4d94ade4-7161-43cf-95dd-90c92da670b0\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") "
Sep 16 13:01:25.232670 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.232662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d94ade4-7161-43cf-95dd-90c92da670b0-kserve-provision-location\") pod \"4d94ade4-7161-43cf-95dd-90c92da670b0\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") "
Sep 16 13:01:25.232954 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.232726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d94ade4-7161-43cf-95dd-90c92da670b0-proxy-tls\") pod \"4d94ade4-7161-43cf-95dd-90c92da670b0\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") "
Sep 16 13:01:25.232954 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.232851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-rbac-proxy-sar-config\") pod \"4d94ade4-7161-43cf-95dd-90c92da670b0\" (UID: \"4d94ade4-7161-43cf-95dd-90c92da670b0\") "
Sep 16 13:01:25.233073 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.232991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d94ade4-7161-43cf-95dd-90c92da670b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d94ade4-7161-43cf-95dd-90c92da670b0" (UID: "4d94ade4-7161-43cf-95dd-90c92da670b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:25.233073 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.233016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d94ade4-7161-43cf-95dd-90c92da670b0" (UID: "4d94ade4-7161-43cf-95dd-90c92da670b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:25.233227 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.233206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d94ade4-7161-43cf-95dd-90c92da670b0" (UID: "4d94ade4-7161-43cf-95dd-90c92da670b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:25.233312 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.233226    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.233312 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.233241    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d94ade4-7161-43cf-95dd-90c92da670b0-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.235603 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.235579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-api-access-vnwqh" (OuterVolumeSpecName: "kube-api-access-vnwqh") pod "4d94ade4-7161-43cf-95dd-90c92da670b0" (UID: "4d94ade4-7161-43cf-95dd-90c92da670b0"). InnerVolumeSpecName "kube-api-access-vnwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:25.235695 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.235596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d94ade4-7161-43cf-95dd-90c92da670b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d94ade4-7161-43cf-95dd-90c92da670b0" (UID: "4d94ade4-7161-43cf-95dd-90c92da670b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:01:25.333827 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.333802    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.333827 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.333824    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwqh\" (UniqueName: \"kubernetes.io/projected/4d94ade4-7161-43cf-95dd-90c92da670b0-kube-api-access-vnwqh\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.333969 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.333833    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d94ade4-7161-43cf-95dd-90c92da670b0-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:25.406667 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.406638    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerID="b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e" exitCode=0
Sep 16 13:01:25.406779 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.406701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"
Sep 16 13:01:25.406779 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.406738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerDied","Data":"b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e"}
Sep 16 13:01:25.406920 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.406844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z" event={"ID":"4d94ade4-7161-43cf-95dd-90c92da670b0","Type":"ContainerDied","Data":"19be7a9c956c8ec3fe6a10ecb934db4066ebbe22809d923087b22f1c55359c80"}
Sep 16 13:01:25.406920 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.406889    2577 scope.go:117] "RemoveContainer" containerID="0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3"
Sep 16 13:01:25.416430 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.416414    2577 scope.go:117] "RemoveContainer" containerID="b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e"
Sep 16 13:01:25.423985 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.423968    2577 scope.go:117] "RemoveContainer" containerID="4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad"
Sep 16 13:01:25.431296 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.431280    2577 scope.go:117] "RemoveContainer" containerID="0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3"
Sep 16 13:01:25.431549 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:25.431534    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3\": container with ID starting with 0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3 not found: ID does not exist" containerID="0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3"
Sep 16 13:01:25.431593 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.431557    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3"} err="failed to get container status \"0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3\": rpc error: code = NotFound desc = could not find container \"0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3\": container with ID starting with 0a7b7f22da1b6dfc63c21263aca21f2d1333a531c3be434ac5f57fdfb7daada3 not found: ID does not exist"
Sep 16 13:01:25.431593 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.431575    2577 scope.go:117] "RemoveContainer" containerID="b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e"
Sep 16 13:01:25.431815 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:25.431798    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e\": container with ID starting with b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e not found: ID does not exist" containerID="b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e"
Sep 16 13:01:25.431882 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.431825    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e"} err="failed to get container status \"b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e\": rpc error: code = NotFound desc = could not find container \"b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e\": container with ID starting with b31cc2736bf305c3c26a7251d7a74c8b3d18701c14f2076f1c0906a40d98aa4e not found: ID does not exist"
Sep 16 13:01:25.431882 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.431842    2577 scope.go:117] "RemoveContainer" containerID="4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad"
Sep 16 13:01:25.432112 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:25.432096    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad\": container with ID starting with 4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad not found: ID does not exist" containerID="4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad"
Sep 16 13:01:25.432168 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.432121    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad"} err="failed to get container status \"4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad\": rpc error: code = NotFound desc = could not find container \"4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad\": container with ID starting with 4734f74d08635cd264f551901004dd0f3b081a0db30253fe3c32ac6ace8c28ad not found: ID does not exist"
Sep 16 13:01:25.436258 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.436236    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"]
Sep 16 13:01:25.445555 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:25.445537    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-zhm5z"]
Sep 16 13:01:26.341644 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.341616    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"]
Sep 16 13:01:26.342046 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342034    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kube-rbac-proxy"
Sep 16 13:01:26.342089 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kube-rbac-proxy"
Sep 16 13:01:26.342089 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342065    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="storage-initializer"
Sep 16 13:01:26.342089 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342071    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="storage-initializer"
Sep 16 13:01:26.342089 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container"
Sep 16 13:01:26.342089 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342090    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container"
Sep 16 13:01:26.342238 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342156    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kserve-container"
Sep 16 13:01:26.342238 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.342167    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" containerName="kube-rbac-proxy"
Sep 16 13:01:26.345544 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.345519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.348744 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.348722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 13:01:26.349039 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.349010    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 13:01:26.364242 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.364220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"]
Sep 16 13:01:26.443839 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.443812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.444016 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.443872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.444016 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.443920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z99v\" (UniqueName: \"kubernetes.io/projected/d427f676-225f-4e4f-abf0-4403420eeccb-kube-api-access-8z99v\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.444016 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.443989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d427f676-225f-4e4f-abf0-4403420eeccb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.444146 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.444060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.544988 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.544955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545136 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.544990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545136 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.545012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z99v\" (UniqueName: \"kubernetes.io/projected/d427f676-225f-4e4f-abf0-4403420eeccb-kube-api-access-8z99v\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545136 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.545034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d427f676-225f-4e4f-abf0-4403420eeccb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545136 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.545069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545136 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:26.545121    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 13:01:26.545388 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:26.545192    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls podName:d427f676-225f-4e4f-abf0-4403420eeccb nodeName:}" failed. No retries permitted until 2026-09-16 13:01:27.045168112 +0000 UTC m=+814.995513650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls") pod "isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" (UID: "d427f676-225f-4e4f-abf0-4403420eeccb") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 13:01:26.545478 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.545460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d427f676-225f-4e4f-abf0-4403420eeccb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545692 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.545676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.545844 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.545825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.554490 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.554460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z99v\" (UniqueName: \"kubernetes.io/projected/d427f676-225f-4e4f-abf0-4403420eeccb-kube-api-access-8z99v\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:26.572748 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:26.572726    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d94ade4-7161-43cf-95dd-90c92da670b0" path="/var/lib/kubelet/pods/4d94ade4-7161-43cf-95dd-90c92da670b0/volumes"
Sep 16 13:01:27.050607 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:27.050570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:27.053330 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:27.053305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-4k5mm\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:27.257144 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:27.257110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:27.540184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:27.540159    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"]
Sep 16 13:01:27.542067 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:01:27.542037    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd427f676_225f_4e4f_abf0_4403420eeccb.slice/crio-059a2dccf7ee5fcf720d10fb2eec00929d289e392077f44b8d30ba61e31f2260 WatchSource:0}: Error finding container 059a2dccf7ee5fcf720d10fb2eec00929d289e392077f44b8d30ba61e31f2260: Status 404 returned error can't find the container with id 059a2dccf7ee5fcf720d10fb2eec00929d289e392077f44b8d30ba61e31f2260
Sep 16 13:01:28.423555 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:28.423520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerStarted","Data":"488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a"}
Sep 16 13:01:28.423746 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:28.423577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerStarted","Data":"059a2dccf7ee5fcf720d10fb2eec00929d289e392077f44b8d30ba61e31f2260"}
Sep 16 13:01:32.443128 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:32.443096    2577 generic.go:356] "Generic (PLEG): container finished" podID="d427f676-225f-4e4f-abf0-4403420eeccb" containerID="488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a" exitCode=0
Sep 16 13:01:32.443634 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:32.443155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerDied","Data":"488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a"}
Sep 16 13:01:33.451496 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:33.451463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerStarted","Data":"2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40"}
Sep 16 13:01:33.451897 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:33.451528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerStarted","Data":"7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567"}
Sep 16 13:01:33.451897 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:33.451724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:33.488506 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:33.488462    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podStartSLOduration=3.575213359 podStartE2EDuration="7.488449675s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913236316s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:33.486342573 +0000 UTC m=+821.436688126" watchObservedRunningTime="2026-09-16 13:01:33.488449675 +0000 UTC m=+821.438795228"
Sep 16 13:01:34.456507 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:34.456479    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:34.457619 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:34.457600    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:01:35.461480 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:35.461442    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:01:36.533328 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.533294    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"]
Sep 16 13:01:36.533670 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.533514    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" containerID="cri-o://061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60" gracePeriod=30
Sep 16 13:01:36.533670 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.533562    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kube-rbac-proxy" containerID="cri-o://7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500" gracePeriod=30
Sep 16 13:01:36.977659 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.977627    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"]
Sep 16 13:01:36.981356 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.981339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:36.985251 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.985220    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-5b43d-kube-rbac-proxy-sar-config\""
Sep 16 13:01:36.990670 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:36.990652    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-5b43d-predictor-serving-cert\""
Sep 16 13:01:37.011160 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.011137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"]
Sep 16 13:01:37.137742 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.137710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51cd9889-bb21-4981-a361-778614b6337a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.137900 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.137767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nst\" (UniqueName: \"kubernetes.io/projected/51cd9889-bb21-4981-a361-778614b6337a-kube-api-access-52nst\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.137900 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.137835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.137900 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.137890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.138024 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.137919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51cd9889-bb21-4981-a361-778614b6337a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.238729 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.238642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51cd9889-bb21-4981-a361-778614b6337a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.238875 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.238726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nst\" (UniqueName: \"kubernetes.io/projected/51cd9889-bb21-4981-a361-778614b6337a-kube-api-access-52nst\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.238875 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.238789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.238875 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.238841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.239067 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.238893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51cd9889-bb21-4981-a361-778614b6337a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.239067 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.239025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51cd9889-bb21-4981-a361-778614b6337a-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.239483 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.239458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.239594 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.239562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.241649 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.241626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51cd9889-bb21-4981-a361-778614b6337a-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.249302 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.249282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nst\" (UniqueName: \"kubernetes.io/projected/51cd9889-bb21-4981-a361-778614b6337a-kube-api-access-52nst\") pod \"isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.291827 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.291796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:01:37.473829 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.473799    2577 generic.go:356] "Generic (PLEG): container finished" podID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerID="7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500" exitCode=2
Sep 16 13:01:37.473994 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.473887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerDied","Data":"7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500"}
Sep 16 13:01:37.587704 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.587681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"]
Sep 16 13:01:37.590312 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:01:37.590285    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cd9889_bb21_4981_a361_778614b6337a.slice/crio-df68ead498dec62f87b7c29963c4dc17f17f3deebae82b4ae285e3226e6c0463 WatchSource:0}: Error finding container df68ead498dec62f87b7c29963c4dc17f17f3deebae82b4ae285e3226e6c0463: Status 404 returned error can't find the container with id df68ead498dec62f87b7c29963c4dc17f17f3deebae82b4ae285e3226e6c0463
Sep 16 13:01:37.976521 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.976489    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 13:01:37.980955 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:37.980924    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 13:01:38.480659 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:38.480624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerStarted","Data":"2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c"}
Sep 16 13:01:38.480817 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:38.480686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerStarted","Data":"df68ead498dec62f87b7c29963c4dc17f17f3deebae82b4ae285e3226e6c0463"}
Sep 16 13:01:40.466034 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:40.466008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:01:40.466485 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:40.466461    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:01:41.085524 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.085501    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 13:01:41.278184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278155    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-rbac-proxy-sar-config\") pod \"9413682b-75bf-46ba-a4e8-80329f35bf1f\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") "
Sep 16 13:01:41.278356 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9413682b-75bf-46ba-a4e8-80329f35bf1f-proxy-tls\") pod \"9413682b-75bf-46ba-a4e8-80329f35bf1f\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") "
Sep 16 13:01:41.278356 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278261    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-cabundle-cert\") pod \"9413682b-75bf-46ba-a4e8-80329f35bf1f\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") "
Sep 16 13:01:41.278356 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c26x\" (UniqueName: \"kubernetes.io/projected/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-api-access-9c26x\") pod \"9413682b-75bf-46ba-a4e8-80329f35bf1f\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") "
Sep 16 13:01:41.278356 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9413682b-75bf-46ba-a4e8-80329f35bf1f-kserve-provision-location\") pod \"9413682b-75bf-46ba-a4e8-80329f35bf1f\" (UID: \"9413682b-75bf-46ba-a4e8-80329f35bf1f\") "
Sep 16 13:01:41.278581 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278532    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9413682b-75bf-46ba-a4e8-80329f35bf1f" (UID: "9413682b-75bf-46ba-a4e8-80329f35bf1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:41.278740 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278714    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9413682b-75bf-46ba-a4e8-80329f35bf1f" (UID: "9413682b-75bf-46ba-a4e8-80329f35bf1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:41.278810 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278739    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:41.278810 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.278755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9413682b-75bf-46ba-a4e8-80329f35bf1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9413682b-75bf-46ba-a4e8-80329f35bf1f" (UID: "9413682b-75bf-46ba-a4e8-80329f35bf1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:41.280404 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.280381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9413682b-75bf-46ba-a4e8-80329f35bf1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9413682b-75bf-46ba-a4e8-80329f35bf1f" (UID: "9413682b-75bf-46ba-a4e8-80329f35bf1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:01:41.280503 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.280480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-api-access-9c26x" (OuterVolumeSpecName: "kube-api-access-9c26x") pod "9413682b-75bf-46ba-a4e8-80329f35bf1f" (UID: "9413682b-75bf-46ba-a4e8-80329f35bf1f"). InnerVolumeSpecName "kube-api-access-9c26x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:41.380209 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.380179    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9413682b-75bf-46ba-a4e8-80329f35bf1f-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:41.380209 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.380212    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c26x\" (UniqueName: \"kubernetes.io/projected/9413682b-75bf-46ba-a4e8-80329f35bf1f-kube-api-access-9c26x\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:41.380381 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.380222    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9413682b-75bf-46ba-a4e8-80329f35bf1f-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:41.380381 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.380230    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9413682b-75bf-46ba-a4e8-80329f35bf1f-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:01:41.497494 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.497459    2577 generic.go:356] "Generic (PLEG): container finished" podID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerID="061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60" exitCode=0
Sep 16 13:01:41.497918 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.497519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"
Sep 16 13:01:41.497918 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.497548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerDied","Data":"061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60"}
Sep 16 13:01:41.497918 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.497602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp" event={"ID":"9413682b-75bf-46ba-a4e8-80329f35bf1f","Type":"ContainerDied","Data":"b5c8cea5450822086efe660912a8e6e6e6cb86bfb0a8ba5de15b9d021b3389ff"}
Sep 16 13:01:41.497918 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.497639    2577 scope.go:117] "RemoveContainer" containerID="7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500"
Sep 16 13:01:41.499227 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.499206    2577 generic.go:356] "Generic (PLEG): container finished" podID="51cd9889-bb21-4981-a361-778614b6337a" containerID="2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c" exitCode=0
Sep 16 13:01:41.499332 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.499294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerDied","Data":"2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c"}
Sep 16 13:01:41.506833 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.506817    2577 scope.go:117] "RemoveContainer" containerID="061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60"
Sep 16 13:01:41.514364 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.514349    2577 scope.go:117] "RemoveContainer" containerID="bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202"
Sep 16 13:01:41.522783 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.522766    2577 scope.go:117] "RemoveContainer" containerID="7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500"
Sep 16 13:01:41.523117 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:41.523096    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500\": container with ID starting with 7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500 not found: ID does not exist" containerID="7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500"
Sep 16 13:01:41.523184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.523127    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500"} err="failed to get container status \"7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500\": rpc error: code = NotFound desc = could not find container \"7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500\": container with ID starting with 7d863e5345fb1ac781fd98dcfa1922a9a2a2e2b2dfecb7087b8e236713e31500 not found: ID does not exist"
Sep 16 13:01:41.523184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.523145    2577 scope.go:117] "RemoveContainer" containerID="061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60"
Sep 16 13:01:41.523438 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:41.523420    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60\": container with ID starting with 061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60 not found: ID does not exist" containerID="061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60"
Sep 16 13:01:41.523488 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.523444    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60"} err="failed to get container status \"061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60\": rpc error: code = NotFound desc = could not find container \"061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60\": container with ID starting with 061dfb76e4315b9ffbd417cb632029663bc16d50f7c8bdeaacb73819b7333f60 not found: ID does not exist"
Sep 16 13:01:41.523488 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.523462    2577 scope.go:117] "RemoveContainer" containerID="bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202"
Sep 16 13:01:41.523697 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:01:41.523681    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202\": container with ID starting with bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202 not found: ID does not exist" containerID="bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202"
Sep 16 13:01:41.523749 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.523706    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202"} err="failed to get container status \"bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202\": rpc error: code = NotFound desc = could not find container \"bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202\": container with ID starting with bb5c975f3f4660de814dda747f91a9360ff571894a290ce2be8baecacb2a8202 not found: ID does not exist"
Sep 16 13:01:41.536634 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.536609    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"]
Sep 16 13:01:41.541967 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:41.541941    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-ec80e-predictor-55c57d89d4-mp2qp"]
Sep 16 13:01:42.576713 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:42.576676    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" path="/var/lib/kubelet/pods/9413682b-75bf-46ba-a4e8-80329f35bf1f/volumes"
Sep 16 13:01:50.466617 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:01:50.466576    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:02:00.466911 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.466851    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:02:00.605916 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.605884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerStarted","Data":"b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a"}
Sep 16 13:02:00.606065 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.605942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerStarted","Data":"bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d"}
Sep 16 13:02:00.606217 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.606201    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:02:00.606376 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.606357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:02:00.607318 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.607295    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:00.628872 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:00.628814    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podStartSLOduration=2.8229961550000002 podStartE2EDuration="24.628795559s" totalImagesPullingTime="18.118971546s" totalInitContainerRuntime="3.686827858s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:00.627720685 +0000 UTC m=+848.578066240" watchObservedRunningTime="2026-09-16 13:02:00.628795559 +0000 UTC m=+848.579141128"
Sep 16 13:02:01.610439 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:01.610395    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:06.615203 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:06.615176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:02:06.615751 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:06.615645    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:10.466801 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:10.466766    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:02:16.616041 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:16.616002    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:20.467025 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:20.466988    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:02:26.616288 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:26.616244    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:30.466511 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:30.466479    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:02:36.616352 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:36.616315    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:40.467257 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:40.467218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 13:02:46.616449 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:46.616362    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 13:02:50.143525 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.143494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"]
Sep 16 13:02:50.143952 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.143724    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container" containerID="cri-o://7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567" gracePeriod=30
Sep 16 13:02:50.143952 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.143786    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kube-rbac-proxy" containerID="cri-o://2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40" gracePeriod=30
Sep 16 13:02:50.144764 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.144743    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:02:50.462208 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.462171    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 13:02:50.851740 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.851659    2577 generic.go:356] "Generic (PLEG): container finished" podID="d427f676-225f-4e4f-abf0-4403420eeccb" containerID="2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40" exitCode=2
Sep 16 13:02:50.851889 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:50.851731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerDied","Data":"2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40"}
Sep 16 13:02:52.537900 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:52.537842    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:02:52.538349 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:52.538011    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:02:54.292094 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.292070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:02:54.422366 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z99v\" (UniqueName: \"kubernetes.io/projected/d427f676-225f-4e4f-abf0-4403420eeccb-kube-api-access-8z99v\") pod \"d427f676-225f-4e4f-abf0-4403420eeccb\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") "
Sep 16 13:02:54.422366 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-kube-rbac-proxy-sar-config\") pod \"d427f676-225f-4e4f-abf0-4403420eeccb\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") "
Sep 16 13:02:54.422366 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d427f676-225f-4e4f-abf0-4403420eeccb-kserve-provision-location\") pod \"d427f676-225f-4e4f-abf0-4403420eeccb\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") "
Sep 16 13:02:54.422614 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls\") pod \"d427f676-225f-4e4f-abf0-4403420eeccb\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") "
Sep 16 13:02:54.422614 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-cabundle-cert\") pod \"d427f676-225f-4e4f-abf0-4403420eeccb\" (UID: \"d427f676-225f-4e4f-abf0-4403420eeccb\") "
Sep 16 13:02:54.422743 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d427f676-225f-4e4f-abf0-4403420eeccb" (UID: "d427f676-225f-4e4f-abf0-4403420eeccb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:02:54.422809 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d427f676-225f-4e4f-abf0-4403420eeccb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d427f676-225f-4e4f-abf0-4403420eeccb" (UID: "d427f676-225f-4e4f-abf0-4403420eeccb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:02:54.422938 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.422917    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d427f676-225f-4e4f-abf0-4403420eeccb" (UID: "d427f676-225f-4e4f-abf0-4403420eeccb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:02:54.424602 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.424570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d427f676-225f-4e4f-abf0-4403420eeccb-kube-api-access-8z99v" (OuterVolumeSpecName: "kube-api-access-8z99v") pod "d427f676-225f-4e4f-abf0-4403420eeccb" (UID: "d427f676-225f-4e4f-abf0-4403420eeccb"). InnerVolumeSpecName "kube-api-access-8z99v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:02:54.424704 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.424637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d427f676-225f-4e4f-abf0-4403420eeccb" (UID: "d427f676-225f-4e4f-abf0-4403420eeccb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:02:54.523906 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.523883    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d427f676-225f-4e4f-abf0-4403420eeccb-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:02:54.523906 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.523903    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:02:54.524038 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.523912    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z99v\" (UniqueName: \"kubernetes.io/projected/d427f676-225f-4e4f-abf0-4403420eeccb-kube-api-access-8z99v\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:02:54.524038 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.523921    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d427f676-225f-4e4f-abf0-4403420eeccb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:02:54.524038 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.523930    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d427f676-225f-4e4f-abf0-4403420eeccb-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:02:54.871551 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.871516    2577 generic.go:356] "Generic (PLEG): container finished" podID="d427f676-225f-4e4f-abf0-4403420eeccb" containerID="7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567" exitCode=0
Sep 16 13:02:54.871687 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.871562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"
Sep 16 13:02:54.871687 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.871638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerDied","Data":"7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567"}
Sep 16 13:02:54.871763 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.871700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm" event={"ID":"d427f676-225f-4e4f-abf0-4403420eeccb","Type":"ContainerDied","Data":"059a2dccf7ee5fcf720d10fb2eec00929d289e392077f44b8d30ba61e31f2260"}
Sep 16 13:02:54.871763 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.871737    2577 scope.go:117] "RemoveContainer" containerID="2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40"
Sep 16 13:02:54.882363 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.882345    2577 scope.go:117] "RemoveContainer" containerID="7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567"
Sep 16 13:02:54.890411 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.890394    2577 scope.go:117] "RemoveContainer" containerID="488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a"
Sep 16 13:02:54.891233 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.891213    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"]
Sep 16 13:02:54.899469 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.899427    2577 scope.go:117] "RemoveContainer" containerID="2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40"
Sep 16 13:02:54.899832 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:02:54.899806    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40\": container with ID starting with 2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40 not found: ID does not exist" containerID="2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40"
Sep 16 13:02:54.899969 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.899844    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40"} err="failed to get container status \"2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40\": rpc error: code = NotFound desc = could not find container \"2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40\": container with ID starting with 2653a02210a8805d9866d3c482fc449db3f9a9ba010457bf0cc2cb01df38dd40 not found: ID does not exist"
Sep 16 13:02:54.899969 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.899893    2577 scope.go:117] "RemoveContainer" containerID="7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567"
Sep 16 13:02:54.900440 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:02:54.900421    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567\": container with ID starting with 7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567 not found: ID does not exist" containerID="7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567"
Sep 16 13:02:54.900504 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.900445    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567"} err="failed to get container status \"7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567\": rpc error: code = NotFound desc = could not find container \"7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567\": container with ID starting with 7684b884410e24a1612fc34a68a45270a6d70968bfd973861a42ef3ee5629567 not found: ID does not exist"
Sep 16 13:02:54.900504 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.900460    2577 scope.go:117] "RemoveContainer" containerID="488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a"
Sep 16 13:02:54.900730 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:02:54.900711    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a\": container with ID starting with 488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a not found: ID does not exist" containerID="488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a"
Sep 16 13:02:54.900792 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.900735    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a"} err="failed to get container status \"488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a\": rpc error: code = NotFound desc = could not find container \"488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a\": container with ID starting with 488e63ce738e333f7a1745a092a736c8a13b7fc237cc950c14de8a1531721e5a not found: ID does not exist"
Sep 16 13:02:54.901849 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:54.901828    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-4k5mm"]
Sep 16 13:02:56.573134 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:56.573099    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" path="/var/lib/kubelet/pods/d427f676-225f-4e4f-abf0-4403420eeccb/volumes"
Sep 16 13:02:56.616067 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:02:56.616042    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:03:27.021359 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"]
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021697    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kube-rbac-proxy"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021714    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kube-rbac-proxy"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021729    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021737    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021751    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kube-rbac-proxy"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021760    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kube-rbac-proxy"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021776    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="storage-initializer"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021781    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="storage-initializer"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021789    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021796    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021806    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="storage-initializer"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021814    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="storage-initializer"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021903    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kserve-container"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021912    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d427f676-225f-4e4f-abf0-4403420eeccb" containerName="kube-rbac-proxy"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021921    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kserve-container"
Sep 16 13:03:27.022943 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.021929    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9413682b-75bf-46ba-a4e8-80329f35bf1f" containerName="kube-rbac-proxy"
Sep 16 13:03:27.023792 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.023775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.026368 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.026348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-6a607-predictor-serving-cert\""
Sep 16 13:03:27.026767 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.026746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-6a607-kube-rbac-proxy-sar-config\""
Sep 16 13:03:27.044527 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.044481    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"]
Sep 16 13:03:27.088645 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.088620    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"]
Sep 16 13:03:27.088896 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.088835    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container" containerID="cri-o://bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d" gracePeriod=30
Sep 16 13:03:27.088978 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.088928    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kube-rbac-proxy" containerID="cri-o://b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a" gracePeriod=30
Sep 16 13:03:27.096472 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.096444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.096570 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.096525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45k6\" (UniqueName: \"kubernetes.io/projected/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-api-access-w45k6\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.096570 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.096557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcac34b4-3a48-4464-9184-b31dead8dfa5-proxy-tls\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.198033 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.198008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.198169 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.198080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45k6\" (UniqueName: \"kubernetes.io/projected/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-api-access-w45k6\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.198169 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.198105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcac34b4-3a48-4464-9184-b31dead8dfa5-proxy-tls\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.198705 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.198684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.200368 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.200352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcac34b4-3a48-4464-9184-b31dead8dfa5-proxy-tls\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.216180 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.216157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45k6\" (UniqueName: \"kubernetes.io/projected/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-api-access-w45k6\") pod \"message-dumper-raw-6a607-predictor-77b769f464-rnm7w\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") " pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.335434 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.335375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:27.618683 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.618617    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"]
Sep 16 13:03:27.621670 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:03:27.621647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcac34b4_3a48_4464_9184_b31dead8dfa5.slice/crio-536af202c794c79136f5974827027fdbca1fd96eb378b1d5c1bd5edae3e80794 WatchSource:0}: Error finding container 536af202c794c79136f5974827027fdbca1fd96eb378b1d5c1bd5edae3e80794: Status 404 returned error can't find the container with id 536af202c794c79136f5974827027fdbca1fd96eb378b1d5c1bd5edae3e80794
Sep 16 13:03:27.623605 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:27.623588    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:03:28.030164 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:28.030132    2577 generic.go:356] "Generic (PLEG): container finished" podID="51cd9889-bb21-4981-a361-778614b6337a" containerID="b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a" exitCode=2
Sep 16 13:03:28.030532 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:28.030205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerDied","Data":"b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a"}
Sep 16 13:03:28.031404 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:28.031381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" event={"ID":"fcac34b4-3a48-4464-9184-b31dead8dfa5","Type":"ContainerStarted","Data":"536af202c794c79136f5974827027fdbca1fd96eb378b1d5c1bd5edae3e80794"}
Sep 16 13:03:29.037772 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:29.037739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" event={"ID":"fcac34b4-3a48-4464-9184-b31dead8dfa5","Type":"ContainerStarted","Data":"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"}
Sep 16 13:03:29.038161 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:29.037805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" event={"ID":"fcac34b4-3a48-4464-9184-b31dead8dfa5","Type":"ContainerStarted","Data":"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"}
Sep 16 13:03:29.058784 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:29.058737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" podStartSLOduration=0.856313647 podStartE2EDuration="2.058721586s" totalImagesPullingTime="1.202407939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:29.057411184 +0000 UTC m=+937.007756749" watchObservedRunningTime="2026-09-16 13:03:29.058721586 +0000 UTC m=+937.009067139"
Sep 16 13:03:30.042913 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.042882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:30.043266 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.042947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:30.044571 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.044544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:03:30.329949 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.329929    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:03:30.427750 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.427726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-cabundle-cert\") pod \"51cd9889-bb21-4981-a361-778614b6337a\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") "
Sep 16 13:03:30.427878 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.427760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-kube-rbac-proxy-sar-config\") pod \"51cd9889-bb21-4981-a361-778614b6337a\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") "
Sep 16 13:03:30.427937 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.427908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51cd9889-bb21-4981-a361-778614b6337a-kserve-provision-location\") pod \"51cd9889-bb21-4981-a361-778614b6337a\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") "
Sep 16 13:03:30.427982 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.427956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51cd9889-bb21-4981-a361-778614b6337a-proxy-tls\") pod \"51cd9889-bb21-4981-a361-778614b6337a\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") "
Sep 16 13:03:30.428029 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428014    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52nst\" (UniqueName: \"kubernetes.io/projected/51cd9889-bb21-4981-a361-778614b6337a-kube-api-access-52nst\") pod \"51cd9889-bb21-4981-a361-778614b6337a\" (UID: \"51cd9889-bb21-4981-a361-778614b6337a\") "
Sep 16 13:03:30.428093 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51cd9889-bb21-4981-a361-778614b6337a" (UID: "51cd9889-bb21-4981-a361-778614b6337a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:30.428218 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428195    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51cd9889-bb21-4981-a361-778614b6337a" (UID: "51cd9889-bb21-4981-a361-778614b6337a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:03:30.428276 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cd9889-bb21-4981-a361-778614b6337a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51cd9889-bb21-4981-a361-778614b6337a" (UID: "51cd9889-bb21-4981-a361-778614b6337a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:30.428343 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428329    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:03:30.428380 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428348    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51cd9889-bb21-4981-a361-778614b6337a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:03:30.428380 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.428357    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51cd9889-bb21-4981-a361-778614b6337a-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:03:30.430189 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.430169    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cd9889-bb21-4981-a361-778614b6337a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51cd9889-bb21-4981-a361-778614b6337a" (UID: "51cd9889-bb21-4981-a361-778614b6337a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:30.430189 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.430170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cd9889-bb21-4981-a361-778614b6337a-kube-api-access-52nst" (OuterVolumeSpecName: "kube-api-access-52nst") pod "51cd9889-bb21-4981-a361-778614b6337a" (UID: "51cd9889-bb21-4981-a361-778614b6337a"). InnerVolumeSpecName "kube-api-access-52nst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:30.528923 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.528890    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51cd9889-bb21-4981-a361-778614b6337a-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:03:30.528923 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:30.528919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52nst\" (UniqueName: \"kubernetes.io/projected/51cd9889-bb21-4981-a361-778614b6337a-kube-api-access-52nst\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:03:31.049686 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.049656    2577 generic.go:356] "Generic (PLEG): container finished" podID="51cd9889-bb21-4981-a361-778614b6337a" containerID="bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d" exitCode=0
Sep 16 13:03:31.050087 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.049712    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"
Sep 16 13:03:31.050087 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.049748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerDied","Data":"bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d"}
Sep 16 13:03:31.050087 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.049807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c" event={"ID":"51cd9889-bb21-4981-a361-778614b6337a","Type":"ContainerDied","Data":"df68ead498dec62f87b7c29963c4dc17f17f3deebae82b4ae285e3226e6c0463"}
Sep 16 13:03:31.050087 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.049834    2577 scope.go:117] "RemoveContainer" containerID="b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a"
Sep 16 13:03:31.058766 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.058748    2577 scope.go:117] "RemoveContainer" containerID="bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d"
Sep 16 13:03:31.066549 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.066527    2577 scope.go:117] "RemoveContainer" containerID="2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c"
Sep 16 13:03:31.071134 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.071117    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"]
Sep 16 13:03:31.073982 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.073960    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5b43d-predictor-56c9b4ddbf-5mc5c"]
Sep 16 13:03:31.075170 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.075157    2577 scope.go:117] "RemoveContainer" containerID="b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a"
Sep 16 13:03:31.075415 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:03:31.075398    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a\": container with ID starting with b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a not found: ID does not exist" containerID="b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a"
Sep 16 13:03:31.075465 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.075422    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a"} err="failed to get container status \"b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a\": rpc error: code = NotFound desc = could not find container \"b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a\": container with ID starting with b7aea4ed29a5b45bf0d0f642b007b557065d63e6e751955f6418f9c10f877b7a not found: ID does not exist"
Sep 16 13:03:31.075465 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.075439    2577 scope.go:117] "RemoveContainer" containerID="bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d"
Sep 16 13:03:31.075685 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:03:31.075671    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d\": container with ID starting with bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d not found: ID does not exist" containerID="bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d"
Sep 16 13:03:31.075726 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.075689    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d"} err="failed to get container status \"bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d\": rpc error: code = NotFound desc = could not find container \"bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d\": container with ID starting with bee039b29dc06165696775761aac57125b4cb53dfab5561c62605c82152bdb8d not found: ID does not exist"
Sep 16 13:03:31.075726 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.075702    2577 scope.go:117] "RemoveContainer" containerID="2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c"
Sep 16 13:03:31.075903 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:03:31.075888    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c\": container with ID starting with 2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c not found: ID does not exist" containerID="2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c"
Sep 16 13:03:31.075951 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:31.075907    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c"} err="failed to get container status \"2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c\": rpc error: code = NotFound desc = could not find container \"2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c\": container with ID starting with 2dd6d92921e37a7c3fe2e65cf149e536f8b05132861f3b1cdf1f0eaa41e80c6c not found: ID does not exist"
Sep 16 13:03:32.573833 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:32.573800    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51cd9889-bb21-4981-a361-778614b6337a" path="/var/lib/kubelet/pods/51cd9889-bb21-4981-a361-778614b6337a/volumes"
Sep 16 13:03:37.061800 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:03:37.061771    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:05:12.025846 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.025814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-6a607-predictor-77b769f464-rnm7w_fcac34b4-3a48-4464-9184-b31dead8dfa5/kserve-container/0.log"
Sep 16 13:05:12.210191 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.210163    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"]
Sep 16 13:05:12.210414 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.210394    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kserve-container" containerID="cri-o://1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0" gracePeriod=30
Sep 16 13:05:12.210477 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.210416    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kube-rbac-proxy" containerID="cri-o://0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444" gracePeriod=30
Sep 16 13:05:12.309897 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.309811    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"]
Sep 16 13:05:12.310293 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310275    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="storage-initializer"
Sep 16 13:05:12.310353 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310298    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="storage-initializer"
Sep 16 13:05:12.310353 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310313    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kube-rbac-proxy"
Sep 16 13:05:12.310353 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kube-rbac-proxy"
Sep 16 13:05:12.310353 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310336    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container"
Sep 16 13:05:12.310353 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310342    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container"
Sep 16 13:05:12.310584 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310412    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kserve-container"
Sep 16 13:05:12.310584 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.310420    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51cd9889-bb21-4981-a361-778614b6337a" containerName="kube-rbac-proxy"
Sep 16 13:05:12.313634 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.313618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.316361 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.316151    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 13:05:12.316361 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.316163    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-ab64c-kube-rbac-proxy-sar-config\""
Sep 16 13:05:12.316361 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.316198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-ab64c-predictor-serving-cert\""
Sep 16 13:05:12.316589 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.316452    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:05:12.324694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.324625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"]
Sep 16 13:05:12.443248 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.443209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-cabundle-cert\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.443412 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.443341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgpz\" (UniqueName: \"kubernetes.io/projected/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-api-access-2fgpz\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.443412 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.443378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.443523 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.443423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bc8672-651f-43ab-b57b-733d9807a4be-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.443523 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.443491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.468348 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.468325    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:05:12.520989 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.520953    2577 generic.go:356] "Generic (PLEG): container finished" podID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerID="0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444" exitCode=2
Sep 16 13:05:12.520989 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.520980    2577 generic.go:356] "Generic (PLEG): container finished" podID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerID="1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0" exitCode=2
Sep 16 13:05:12.521184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.521018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"
Sep 16 13:05:12.521184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.521029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" event={"ID":"fcac34b4-3a48-4464-9184-b31dead8dfa5","Type":"ContainerDied","Data":"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"}
Sep 16 13:05:12.521184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.521098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" event={"ID":"fcac34b4-3a48-4464-9184-b31dead8dfa5","Type":"ContainerDied","Data":"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"}
Sep 16 13:05:12.521184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.521120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w" event={"ID":"fcac34b4-3a48-4464-9184-b31dead8dfa5","Type":"ContainerDied","Data":"536af202c794c79136f5974827027fdbca1fd96eb378b1d5c1bd5edae3e80794"}
Sep 16 13:05:12.521184 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.521144    2577 scope.go:117] "RemoveContainer" containerID="0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"
Sep 16 13:05:12.529294 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.529276    2577 scope.go:117] "RemoveContainer" containerID="1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"
Sep 16 13:05:12.536590 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.536553    2577 scope.go:117] "RemoveContainer" containerID="0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"
Sep 16 13:05:12.536838 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:05:12.536818    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444\": container with ID starting with 0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444 not found: ID does not exist" containerID="0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"
Sep 16 13:05:12.536925 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.536845    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"} err="failed to get container status \"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444\": rpc error: code = NotFound desc = could not find container \"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444\": container with ID starting with 0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444 not found: ID does not exist"
Sep 16 13:05:12.536925 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.536878    2577 scope.go:117] "RemoveContainer" containerID="1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"
Sep 16 13:05:12.537123 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:05:12.537105    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0\": container with ID starting with 1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0 not found: ID does not exist" containerID="1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"
Sep 16 13:05:12.537173 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.537130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"} err="failed to get container status \"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0\": rpc error: code = NotFound desc = could not find container \"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0\": container with ID starting with 1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0 not found: ID does not exist"
Sep 16 13:05:12.537173 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.537143    2577 scope.go:117] "RemoveContainer" containerID="0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"
Sep 16 13:05:12.537420 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.537403    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444"} err="failed to get container status \"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444\": rpc error: code = NotFound desc = could not find container \"0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444\": container with ID starting with 0e5dfea245fb8014f393feaefccbc10a98a214d1ac50d15e1e969893a9f67444 not found: ID does not exist"
Sep 16 13:05:12.537420 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.537421    2577 scope.go:117] "RemoveContainer" containerID="1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"
Sep 16 13:05:12.537695 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.537672    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0"} err="failed to get container status \"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0\": rpc error: code = NotFound desc = could not find container \"1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0\": container with ID starting with 1b3ffeab3bf84b60a185239d2404a48b69360b237e94711aca829cff042a37c0 not found: ID does not exist"
Sep 16 13:05:12.544156 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-cabundle-cert\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544250 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgpz\" (UniqueName: \"kubernetes.io/projected/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-api-access-2fgpz\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544250 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544372 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bc8672-651f-43ab-b57b-733d9807a4be-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544372 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544479 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:05:12.544381    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-serving-cert: secret "isvc-sklearn-scale-raw-ab64c-predictor-serving-cert" not found
Sep 16 13:05:12.544479 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:05:12.544455    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls podName:a7bc8672-651f-43ab-b57b-733d9807a4be nodeName:}" failed. No retries permitted until 2026-09-16 13:05:13.044434284 +0000 UTC m=+1040.994779817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls") pod "isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" (UID: "a7bc8672-651f-43ab-b57b-733d9807a4be") : secret "isvc-sklearn-scale-raw-ab64c-predictor-serving-cert" not found
Sep 16 13:05:12.544717 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bc8672-651f-43ab-b57b-733d9807a4be-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544923 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-cabundle-cert\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.544923 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.544914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.554592 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.554573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgpz\" (UniqueName: \"kubernetes.io/projected/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-api-access-2fgpz\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:12.645402 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.645350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w45k6\" (UniqueName: \"kubernetes.io/projected/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-api-access-w45k6\") pod \"fcac34b4-3a48-4464-9184-b31dead8dfa5\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") "
Sep 16 13:05:12.645402 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.645383    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-rbac-proxy-sar-config\") pod \"fcac34b4-3a48-4464-9184-b31dead8dfa5\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") "
Sep 16 13:05:12.645567 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.645414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcac34b4-3a48-4464-9184-b31dead8dfa5-proxy-tls\") pod \"fcac34b4-3a48-4464-9184-b31dead8dfa5\" (UID: \"fcac34b4-3a48-4464-9184-b31dead8dfa5\") "
Sep 16 13:05:12.645716 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.645692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcac34b4-3a48-4464-9184-b31dead8dfa5" (UID: "fcac34b4-3a48-4464-9184-b31dead8dfa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:12.647511 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.647491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-api-access-w45k6" (OuterVolumeSpecName: "kube-api-access-w45k6") pod "fcac34b4-3a48-4464-9184-b31dead8dfa5" (UID: "fcac34b4-3a48-4464-9184-b31dead8dfa5"). InnerVolumeSpecName "kube-api-access-w45k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:12.647575 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.647506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcac34b4-3a48-4464-9184-b31dead8dfa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcac34b4-3a48-4464-9184-b31dead8dfa5" (UID: "fcac34b4-3a48-4464-9184-b31dead8dfa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:12.746563 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.746535    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:05:12.746563 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.746555    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcac34b4-3a48-4464-9184-b31dead8dfa5-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:05:12.746563 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.746565    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45k6\" (UniqueName: \"kubernetes.io/projected/fcac34b4-3a48-4464-9184-b31dead8dfa5-kube-api-access-w45k6\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:05:12.841746 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.841723    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"]
Sep 16 13:05:12.844910 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:12.844886    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6a607-predictor-77b769f464-rnm7w"]
Sep 16 13:05:13.048958 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:13.048931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:13.051360 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:13.051342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls\") pod \"isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:13.228139 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:13.228116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:13.503310 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:13.503283    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"]
Sep 16 13:05:13.506212 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:05:13.506188    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bc8672_651f_43ab_b57b_733d9807a4be.slice/crio-e1c79577de01ffb7577611d38ed1ca6f7b7cf315dd89c82baa0d70af165c21d4 WatchSource:0}: Error finding container e1c79577de01ffb7577611d38ed1ca6f7b7cf315dd89c82baa0d70af165c21d4: Status 404 returned error can't find the container with id e1c79577de01ffb7577611d38ed1ca6f7b7cf315dd89c82baa0d70af165c21d4
Sep 16 13:05:13.527330 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:13.527307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerStarted","Data":"e1c79577de01ffb7577611d38ed1ca6f7b7cf315dd89c82baa0d70af165c21d4"}
Sep 16 13:05:14.533409 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:14.533377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerStarted","Data":"b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0"}
Sep 16 13:05:14.572616 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:14.572577    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" path="/var/lib/kubelet/pods/fcac34b4-3a48-4464-9184-b31dead8dfa5/volumes"
Sep 16 13:05:17.547447 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:17.547370    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerID="b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0" exitCode=0
Sep 16 13:05:17.547447 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:17.547419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerDied","Data":"b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0"}
Sep 16 13:05:18.554650 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:18.554617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerStarted","Data":"dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff"}
Sep 16 13:05:18.555053 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:18.554672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerStarted","Data":"bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e"}
Sep 16 13:05:18.555053 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:18.554876    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:18.578027 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:18.577983    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podStartSLOduration=2.941378851 podStartE2EDuration="6.577970491s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.63659164s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:18.575978218 +0000 UTC m=+1046.526323771" watchObservedRunningTime="2026-09-16 13:05:18.577970491 +0000 UTC m=+1046.528316120"
Sep 16 13:05:19.559574 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:19.559541    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:19.560827 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:19.560798    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:05:20.564431 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:20.564390    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:05:25.569367 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:25.569331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:05:25.569823 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:25.569795    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:05:32.131681 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.131644    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"]
Sep 16 13:05:32.132281 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.132227    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kserve-container"
Sep 16 13:05:32.132281 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.132250    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kserve-container"
Sep 16 13:05:32.132281 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.132280    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kube-rbac-proxy"
Sep 16 13:05:32.132460 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.132288    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kube-rbac-proxy"
Sep 16 13:05:32.132460 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.132384    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kserve-container"
Sep 16 13:05:32.132460 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.132401    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcac34b4-3a48-4464-9184-b31dead8dfa5" containerName="kube-rbac-proxy"
Sep 16 13:05:32.138459 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.136331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.141348 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.141322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-d792b-predictor-serving-cert\""
Sep 16 13:05:32.142284 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.141355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-d792b-kube-rbac-proxy-sar-config\""
Sep 16 13:05:32.145292 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.145268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"]
Sep 16 13:05:32.201154 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.201130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kserve-provision-location\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.201274 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.201166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-proxy-tls\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.201274 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.201183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2f4\" (UniqueName: \"kubernetes.io/projected/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-api-access-6m2f4\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.201274 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.201198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.201441 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.201297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-cabundle-cert\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.301736 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.301714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kserve-provision-location\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.301847 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.301749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-proxy-tls\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.301847 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.301765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2f4\" (UniqueName: \"kubernetes.io/projected/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-api-access-6m2f4\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.301847 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.301782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.301847 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.301831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-cabundle-cert\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.302296 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.302264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kserve-provision-location\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.302531 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.302491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-cabundle-cert\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.302807 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.302783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.304484 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.304459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-proxy-tls\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.309798 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.309773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2f4\" (UniqueName: \"kubernetes.io/projected/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-api-access-6m2f4\") pod \"raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") " pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.452451 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.452428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:32.724171 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:32.724151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"]
Sep 16 13:05:32.726237 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:05:32.726216    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6d5f8c_9e7c_4216_9713_d2c2711bdd1e.slice/crio-d1ca0d94e085224cd0d8e6b4a764d702624f02702cc4d349cb4289da589c3d86 WatchSource:0}: Error finding container d1ca0d94e085224cd0d8e6b4a764d702624f02702cc4d349cb4289da589c3d86: Status 404 returned error can't find the container with id d1ca0d94e085224cd0d8e6b4a764d702624f02702cc4d349cb4289da589c3d86
Sep 16 13:05:33.623966 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:33.623933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerStarted","Data":"20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f"}
Sep 16 13:05:33.624347 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:33.623990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerStarted","Data":"d1ca0d94e085224cd0d8e6b4a764d702624f02702cc4d349cb4289da589c3d86"}
Sep 16 13:05:35.570386 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:35.570345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:05:36.645200 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:36.645169    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerID="20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f" exitCode=0
Sep 16 13:05:36.645506 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:36.645214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerDied","Data":"20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f"}
Sep 16 13:05:37.652270 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:37.652236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerStarted","Data":"353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e"}
Sep 16 13:05:37.652672 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:37.652297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerStarted","Data":"ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8"}
Sep 16 13:05:37.652672 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:37.652496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:37.670590 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:37.670542    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podStartSLOduration=1.8818855490000002 podStartE2EDuration="5.670523604s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788638055s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:37.670509139 +0000 UTC m=+1065.620854725" watchObservedRunningTime="2026-09-16 13:05:37.670523604 +0000 UTC m=+1065.620869194"
Sep 16 13:05:38.657350 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:38.657320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:38.658551 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:38.658528    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:05:39.662641 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:39.662598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:05:44.667367 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:44.667292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:05:44.667801 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:44.667776    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:05:45.570090 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:45.570044    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:05:54.668029 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:54.667992    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:05:55.569767 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:05:55.569731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:04.668303 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:04.668264    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:06:05.569818 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:05.569779    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:14.668083 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:14.668047    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:06:15.570421 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:15.570385    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:24.668319 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:24.668275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:06:25.569754 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:25.569719    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:34.668212 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:34.668169    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:06:35.570600 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:35.570563    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:44.667917 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:44.667879    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:06:45.570559 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:45.570525    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:51.567911 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:51.567874    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:06:54.668393 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:06:54.668366    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:07:01.567932 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:01.567894    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:07:02.254219 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:02.254179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"]
Sep 16 13:07:02.254530 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:02.254473    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" containerID="cri-o://ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8" gracePeriod=30
Sep 16 13:07:02.254668 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:02.254562    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kube-rbac-proxy" containerID="cri-o://353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e" gracePeriod=30
Sep 16 13:07:03.058221 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:03.058178    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerID="353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e" exitCode=2
Sep 16 13:07:03.058657 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:03.058249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerDied","Data":"353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e"}
Sep 16 13:07:04.663620 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:04.663584    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 16 13:07:04.667936 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:04.667909    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 13:07:06.497085 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.497061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:07:06.586740 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.586714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-cabundle-cert\") pod \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") "
Sep 16 13:07:06.586913 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.586764    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-proxy-tls\") pod \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") "
Sep 16 13:07:06.586913 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.586779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-rbac-proxy-sar-config\") pod \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") "
Sep 16 13:07:06.586913 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.586829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kserve-provision-location\") pod \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") "
Sep 16 13:07:06.586913 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.586896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m2f4\" (UniqueName: \"kubernetes.io/projected/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-api-access-6m2f4\") pod \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\" (UID: \"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e\") "
Sep 16 13:07:06.587175 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.587137    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" (UID: "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:07:06.587175 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.587160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" (UID: "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:06.587319 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.587194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" (UID: "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:07:06.589109 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.589056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-api-access-6m2f4" (OuterVolumeSpecName: "kube-api-access-6m2f4") pod "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" (UID: "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e"). InnerVolumeSpecName "kube-api-access-6m2f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:07:06.589165 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.589126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" (UID: "fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:07:06.687635 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.687611    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2f4\" (UniqueName: \"kubernetes.io/projected/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-api-access-6m2f4\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:06.687635 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.687633    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:06.687767 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.687643    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:06.687767 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.687650    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:06.687767 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:06.687658    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:07.078459 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.078425    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerID="ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8" exitCode=0
Sep 16 13:07:07.078621 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.078485    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"
Sep 16 13:07:07.078621 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.078509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerDied","Data":"ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8"}
Sep 16 13:07:07.078621 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.078573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd" event={"ID":"fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e","Type":"ContainerDied","Data":"d1ca0d94e085224cd0d8e6b4a764d702624f02702cc4d349cb4289da589c3d86"}
Sep 16 13:07:07.078621 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.078606    2577 scope.go:117] "RemoveContainer" containerID="353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e"
Sep 16 13:07:07.087561 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.087543    2577 scope.go:117] "RemoveContainer" containerID="ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8"
Sep 16 13:07:07.095057 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.095040    2577 scope.go:117] "RemoveContainer" containerID="20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f"
Sep 16 13:07:07.099318 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.099298    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"]
Sep 16 13:07:07.102888 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.102851    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d792b-predictor-6fcd8cd5dd-ww4jd"]
Sep 16 13:07:07.103433 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.103418    2577 scope.go:117] "RemoveContainer" containerID="353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e"
Sep 16 13:07:07.103703 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:07:07.103684    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e\": container with ID starting with 353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e not found: ID does not exist" containerID="353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e"
Sep 16 13:07:07.103745 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.103712    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e"} err="failed to get container status \"353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e\": rpc error: code = NotFound desc = could not find container \"353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e\": container with ID starting with 353a0d3cc2275565a824c6e5839dfbdd04dffebccc6aaf068f02228b16607b5e not found: ID does not exist"
Sep 16 13:07:07.103745 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.103730    2577 scope.go:117] "RemoveContainer" containerID="ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8"
Sep 16 13:07:07.104018 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:07:07.103999    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8\": container with ID starting with ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8 not found: ID does not exist" containerID="ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8"
Sep 16 13:07:07.104094 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.104028    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8"} err="failed to get container status \"ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8\": rpc error: code = NotFound desc = could not find container \"ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8\": container with ID starting with ec2f1c54270cdb935b88bec351504e360c88d9bbbb0cadddcdcb635f66a4a7d8 not found: ID does not exist"
Sep 16 13:07:07.104094 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.104049    2577 scope.go:117] "RemoveContainer" containerID="20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f"
Sep 16 13:07:07.104325 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:07:07.104299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f\": container with ID starting with 20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f not found: ID does not exist" containerID="20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f"
Sep 16 13:07:07.104366 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.104331    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f"} err="failed to get container status \"20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f\": rpc error: code = NotFound desc = could not find container \"20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f\": container with ID starting with 20f9faf6783f7fc997446df28b6a4250a22baf216b7004c10d45fc2626a9340f not found: ID does not exist"
Sep 16 13:07:07.421220 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421149    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"]
Sep 16 13:07:07.421530 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421517    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container"
Sep 16 13:07:07.421583 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421531    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container"
Sep 16 13:07:07.421583 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421543    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="storage-initializer"
Sep 16 13:07:07.421583 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421548    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="storage-initializer"
Sep 16 13:07:07.421583 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kube-rbac-proxy"
Sep 16 13:07:07.421583 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421567    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kube-rbac-proxy"
Sep 16 13:07:07.421731 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421628    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kube-rbac-proxy"
Sep 16 13:07:07.421731 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.421636    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" containerName="kserve-container"
Sep 16 13:07:07.425730 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.425707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.429233 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.429212    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-42956-predictor-serving-cert\""
Sep 16 13:07:07.429343 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.429275    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-42956-kube-rbac-proxy-sar-config\""
Sep 16 13:07:07.442651 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.442625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"]
Sep 16 13:07:07.494952 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.494929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7lm\" (UniqueName: \"kubernetes.io/projected/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-api-access-rt7lm\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.495075 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.494986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kserve-provision-location\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.495075 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.495055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.495205 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.495101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-cabundle-cert\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.495205 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.495186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.595692 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.595661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kserve-provision-location\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596139 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.595712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596139 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.595752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-cabundle-cert\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596139 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.595822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596139 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.595888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7lm\" (UniqueName: \"kubernetes.io/projected/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-api-access-rt7lm\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596139 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.596107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kserve-provision-location\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596643 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.596623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-cabundle-cert\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.596686 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.596623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.598228 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.598212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.603697 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.603676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7lm\" (UniqueName: \"kubernetes.io/projected/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-api-access-rt7lm\") pod \"raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:07.735337 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:07.735316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:08.008225 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:08.008201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"]
Sep 16 13:07:08.010772 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:07:08.010726    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9ea824_7e6b_43ea_96b1_eb781a9ce3cd.slice/crio-c638c487eb1bc356b0de5ea9303db33bd9eda98a39d7346c59b4f7122cf43345 WatchSource:0}: Error finding container c638c487eb1bc356b0de5ea9303db33bd9eda98a39d7346c59b4f7122cf43345: Status 404 returned error can't find the container with id c638c487eb1bc356b0de5ea9303db33bd9eda98a39d7346c59b4f7122cf43345
Sep 16 13:07:08.085668 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:08.085636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerStarted","Data":"425e3c5754fc272f90523f9572c609849681dd09bdf4eedf603f8f8bda2c7d61"}
Sep 16 13:07:08.085784 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:08.085702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerStarted","Data":"c638c487eb1bc356b0de5ea9303db33bd9eda98a39d7346c59b4f7122cf43345"}
Sep 16 13:07:08.573768 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:08.573733    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e" path="/var/lib/kubelet/pods/fe6d5f8c-9e7c-4216-9713-d2c2711bdd1e/volumes"
Sep 16 13:07:11.567785 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:11.567733    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:07:12.105938 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:12.105909    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerID="425e3c5754fc272f90523f9572c609849681dd09bdf4eedf603f8f8bda2c7d61" exitCode=0
Sep 16 13:07:12.106074 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:12.105984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerDied","Data":"425e3c5754fc272f90523f9572c609849681dd09bdf4eedf603f8f8bda2c7d61"}
Sep 16 13:07:13.112664 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:13.112632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerStarted","Data":"659944a2f37ec77553342c8a07ea5f4f8d2d65f47c4e7f9a1534212c1f6dd1a6"}
Sep 16 13:07:13.113056 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:13.112688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerStarted","Data":"b5bdc4672e1e86c5012e857f7bbad50f66e67b08c911e2f18445435bd33b8730"}
Sep 16 13:07:13.113056 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:13.112977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:13.113056 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:13.113029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:13.114502 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:13.114475    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:07:14.117876 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:14.117774    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:07:19.122063 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:19.122034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:07:19.122695 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:19.122668    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:07:19.144601 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:19.144554    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podStartSLOduration=8.286725863000001 podStartE2EDuration="12.144539562s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857813699s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:13.139331599 +0000 UTC m=+1161.089677172" watchObservedRunningTime="2026-09-16 13:07:19.144539562 +0000 UTC m=+1167.094885116"
Sep 16 13:07:21.567811 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:21.567765    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:07:29.123072 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:29.123035    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:07:31.568699 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:31.568654    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 13:07:39.123500 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:39.123465    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:07:41.568693 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:41.568663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:07:42.380504 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:42.380473    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"]
Sep 16 13:07:42.380777 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:42.380753    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container" containerID="cri-o://bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e" gracePeriod=30
Sep 16 13:07:42.380899 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:42.380777    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kube-rbac-proxy" containerID="cri-o://dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff" gracePeriod=30
Sep 16 13:07:43.257568 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:43.257538    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerID="dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff" exitCode=2
Sep 16 13:07:43.257953 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:43.257619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerDied","Data":"dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff"}
Sep 16 13:07:45.564650 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:45.564609    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 13:07:49.123054 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:49.123016    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:07:50.565578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:50.565539    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 16 13:07:51.434707 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.434684    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:07:51.472477 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472453    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-rbac-proxy-sar-config\") pod \"a7bc8672-651f-43ab-b57b-733d9807a4be\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") "
Sep 16 13:07:51.472622 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fgpz\" (UniqueName: \"kubernetes.io/projected/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-api-access-2fgpz\") pod \"a7bc8672-651f-43ab-b57b-733d9807a4be\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") "
Sep 16 13:07:51.472622 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472542    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-cabundle-cert\") pod \"a7bc8672-651f-43ab-b57b-733d9807a4be\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") "
Sep 16 13:07:51.472622 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bc8672-651f-43ab-b57b-733d9807a4be-kserve-provision-location\") pod \"a7bc8672-651f-43ab-b57b-733d9807a4be\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") "
Sep 16 13:07:51.472622 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls\") pod \"a7bc8672-651f-43ab-b57b-733d9807a4be\" (UID: \"a7bc8672-651f-43ab-b57b-733d9807a4be\") "
Sep 16 13:07:51.472844 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7bc8672-651f-43ab-b57b-733d9807a4be" (UID: "a7bc8672-651f-43ab-b57b-733d9807a4be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:07:51.472992 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472895    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:51.472992 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7bc8672-651f-43ab-b57b-733d9807a4be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7bc8672-651f-43ab-b57b-733d9807a4be" (UID: "a7bc8672-651f-43ab-b57b-733d9807a4be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:51.472992 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.472954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7bc8672-651f-43ab-b57b-733d9807a4be" (UID: "a7bc8672-651f-43ab-b57b-733d9807a4be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:07:51.474804 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.474775    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7bc8672-651f-43ab-b57b-733d9807a4be" (UID: "a7bc8672-651f-43ab-b57b-733d9807a4be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:07:51.474849 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.474804    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-api-access-2fgpz" (OuterVolumeSpecName: "kube-api-access-2fgpz") pod "a7bc8672-651f-43ab-b57b-733d9807a4be" (UID: "a7bc8672-651f-43ab-b57b-733d9807a4be"). InnerVolumeSpecName "kube-api-access-2fgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:07:51.573558 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.573497    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgpz\" (UniqueName: \"kubernetes.io/projected/a7bc8672-651f-43ab-b57b-733d9807a4be-kube-api-access-2fgpz\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:51.573558 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.573520    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bc8672-651f-43ab-b57b-733d9807a4be-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:51.573558 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.573530    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bc8672-651f-43ab-b57b-733d9807a4be-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:51.573558 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:51.573539    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bc8672-651f-43ab-b57b-733d9807a4be-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:07:52.300019 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.299988    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerID="bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e" exitCode=0
Sep 16 13:07:52.300195 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.300040    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"
Sep 16 13:07:52.300195 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.300059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerDied","Data":"bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e"}
Sep 16 13:07:52.300195 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.300113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw" event={"ID":"a7bc8672-651f-43ab-b57b-733d9807a4be","Type":"ContainerDied","Data":"e1c79577de01ffb7577611d38ed1ca6f7b7cf315dd89c82baa0d70af165c21d4"}
Sep 16 13:07:52.300195 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.300141    2577 scope.go:117] "RemoveContainer" containerID="dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff"
Sep 16 13:07:52.309369 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.309345    2577 scope.go:117] "RemoveContainer" containerID="bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e"
Sep 16 13:07:52.316926 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.316907    2577 scope.go:117] "RemoveContainer" containerID="b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0"
Sep 16 13:07:52.321392 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.321368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"]
Sep 16 13:07:52.324508 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.324489    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-ab64c-predictor-d89cf7546-zl4qw"]
Sep 16 13:07:52.325132 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.325119    2577 scope.go:117] "RemoveContainer" containerID="dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff"
Sep 16 13:07:52.325423 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:07:52.325406    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff\": container with ID starting with dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff not found: ID does not exist" containerID="dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff"
Sep 16 13:07:52.325474 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.325431    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff"} err="failed to get container status \"dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff\": rpc error: code = NotFound desc = could not find container \"dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff\": container with ID starting with dd4e564cdb45ca263e7eb18cad12c012624122b77bd1d35b0175ad2ee7d684ff not found: ID does not exist"
Sep 16 13:07:52.325474 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.325448    2577 scope.go:117] "RemoveContainer" containerID="bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e"
Sep 16 13:07:52.325707 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:07:52.325679    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e\": container with ID starting with bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e not found: ID does not exist" containerID="bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e"
Sep 16 13:07:52.325748 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.325715    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e"} err="failed to get container status \"bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e\": rpc error: code = NotFound desc = could not find container \"bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e\": container with ID starting with bb2adeaa5c78e8b252f5c59d3ecf9030a59715a396443ab93579fa7433a25e4e not found: ID does not exist"
Sep 16 13:07:52.325748 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.325730    2577 scope.go:117] "RemoveContainer" containerID="b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0"
Sep 16 13:07:52.326022 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:07:52.326003    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0\": container with ID starting with b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0 not found: ID does not exist" containerID="b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0"
Sep 16 13:07:52.326089 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.326031    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0"} err="failed to get container status \"b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0\": rpc error: code = NotFound desc = could not find container \"b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0\": container with ID starting with b539979646bd3fbac8bc04887612e4f77a2542b0ece6025ae485561071a9abb0 not found: ID does not exist"
Sep 16 13:07:52.574351 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.574269    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" path="/var/lib/kubelet/pods/a7bc8672-651f-43ab-b57b-733d9807a4be/volumes"
Sep 16 13:07:52.575015 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.574992    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:07:52.575390 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:52.575369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:07:59.123087 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:07:59.123049    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:08:09.123255 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:09.123221    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:08:19.122938 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:19.122898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 13:08:26.572770 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:26.572745    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:08:27.479359 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:27.479328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"]
Sep 16 13:08:27.558093 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:27.558058    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-serving-cert: secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:27.558278 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:27.558147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls podName:8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd nodeName:}" failed. No retries permitted until 2026-09-16 13:08:28.058125816 +0000 UTC m=+1236.008471351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls") pod "raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd") : secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:28.061263 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:28.061239    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-serving-cert: secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:28.061613 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:28.061306    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls podName:8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd nodeName:}" failed. No retries permitted until 2026-09-16 13:08:29.061292654 +0000 UTC m=+1237.011638186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls") pod "raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd") : secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:28.474703 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:28.474668    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container" containerID="cri-o://b5bdc4672e1e86c5012e857f7bbad50f66e67b08c911e2f18445435bd33b8730" gracePeriod=30
Sep 16 13:08:28.474893 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:28.474692    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kube-rbac-proxy" containerID="cri-o://659944a2f37ec77553342c8a07ea5f4f8d2d65f47c4e7f9a1534212c1f6dd1a6" gracePeriod=30
Sep 16 13:08:29.070213 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:29.070181    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-serving-cert: secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:29.070567 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:29.070246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls podName:8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd nodeName:}" failed. No retries permitted until 2026-09-16 13:08:31.070232894 +0000 UTC m=+1239.020578425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls") pod "raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd") : secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:29.118019 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:29.117988    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 16 13:08:29.482231 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:29.482203    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerID="659944a2f37ec77553342c8a07ea5f4f8d2d65f47c4e7f9a1534212c1f6dd1a6" exitCode=2
Sep 16 13:08:29.482411 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:29.482238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerDied","Data":"659944a2f37ec77553342c8a07ea5f4f8d2d65f47c4e7f9a1534212c1f6dd1a6"}
Sep 16 13:08:31.087273 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:31.087241    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-serving-cert: secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:31.087634 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:08:31.087308    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls podName:8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd nodeName:}" failed. No retries permitted until 2026-09-16 13:08:35.08729404 +0000 UTC m=+1243.037639570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls") pod "raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd") : secret "raw-sklearn-runtime-42956-predictor-serving-cert" not found
Sep 16 13:08:32.498421 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.498391    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerID="b5bdc4672e1e86c5012e857f7bbad50f66e67b08c911e2f18445435bd33b8730" exitCode=0
Sep 16 13:08:32.498718 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.498458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerDied","Data":"b5bdc4672e1e86c5012e857f7bbad50f66e67b08c911e2f18445435bd33b8730"}
Sep 16 13:08:32.622766 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.622745    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:08:32.701025 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.700956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt7lm\" (UniqueName: \"kubernetes.io/projected/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-api-access-rt7lm\") pod \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") "
Sep 16 13:08:32.701172 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701047    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kserve-provision-location\") pod \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") "
Sep 16 13:08:32.701172 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls\") pod \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") "
Sep 16 13:08:32.701172 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701118    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-rbac-proxy-sar-config\") pod \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") "
Sep 16 13:08:32.701172 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-cabundle-cert\") pod \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\" (UID: \"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd\") "
Sep 16 13:08:32.701413 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701387    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:08:32.701474 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701425    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:32.701551 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.701526    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:32.703334 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.703313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:08:32.703433 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.703360    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-api-access-rt7lm" (OuterVolumeSpecName: "kube-api-access-rt7lm") pod "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" (UID: "8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd"). InnerVolumeSpecName "kube-api-access-rt7lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:32.802499 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.802478    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:08:32.802499 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.802500    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7lm\" (UniqueName: \"kubernetes.io/projected/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-api-access-rt7lm\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:08:32.802654 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.802509    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:08:32.802654 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.802518    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:08:32.802654 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:32.802527    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:08:33.505035 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.505004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9" event={"ID":"8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd","Type":"ContainerDied","Data":"c638c487eb1bc356b0de5ea9303db33bd9eda98a39d7346c59b4f7122cf43345"}
Sep 16 13:08:33.505035 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.505030    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"
Sep 16 13:08:33.505462 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.505099    2577 scope.go:117] "RemoveContainer" containerID="659944a2f37ec77553342c8a07ea5f4f8d2d65f47c4e7f9a1534212c1f6dd1a6"
Sep 16 13:08:33.514459 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.514444    2577 scope.go:117] "RemoveContainer" containerID="b5bdc4672e1e86c5012e857f7bbad50f66e67b08c911e2f18445435bd33b8730"
Sep 16 13:08:33.521941 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.521925    2577 scope.go:117] "RemoveContainer" containerID="425e3c5754fc272f90523f9572c609849681dd09bdf4eedf603f8f8bda2c7d61"
Sep 16 13:08:33.529507 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.529492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"]
Sep 16 13:08:33.532833 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:33.532814    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-42956-predictor-7f4d4c4976-fv7r9"]
Sep 16 13:08:34.572617 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:08:34.572583    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" path="/var/lib/kubelet/pods/8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd/volumes"
Sep 16 13:09:22.848565 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.848518    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"]
Sep 16 13:09:22.850121 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850091    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container"
Sep 16 13:09:22.850121 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850122    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850131    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="storage-initializer"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850140    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="storage-initializer"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850195    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850244    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850265    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kube-rbac-proxy"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850273    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kube-rbac-proxy"
Sep 16 13:09:22.850329 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kube-rbac-proxy"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850339    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kube-rbac-proxy"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850363    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="storage-initializer"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850373    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="storage-initializer"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850478    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kserve-container"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850491    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7bc8672-651f-43ab-b57b-733d9807a4be" containerName="kube-rbac-proxy"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850501    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kserve-container"
Sep 16 13:09:22.850694 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.850509    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ea824-7e6b-43ea-96b1-eb781a9ce3cd" containerName="kube-rbac-proxy"
Sep 16 13:09:22.855213 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.855187    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"]
Sep 16 13:09:22.856610 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.856561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:22.860078 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.859948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 13:09:22.860078 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.860000    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 13:09:22.860324 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.860292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9sk5b\""
Sep 16 13:09:22.860609 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.860591    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 13:09:22.861151 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.861118    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 13:09:22.861311 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.861209    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:09:22.861311 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.861119    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 13:09:22.861612 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.861593    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"]
Sep 16 13:09:22.861754 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.861735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:22.861830 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.861763    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"]
Sep 16 13:09:22.864053 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.864031    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 13:09:22.986044 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/705e22ba-898d-48aa-b4d9-87819af7098f-kube-api-access-qmnbl\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:22.986211 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/705e22ba-898d-48aa-b4d9-87819af7098f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:22.986211 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:22.986211 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:22.986360 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:22.986360 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c651075f-f8e6-4183-b9cd-585f87e95cf8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:22.986360 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:22.986360 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njx5k\" (UniqueName: \"kubernetes.io/projected/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-api-access-njx5k\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:22.986360 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/705e22ba-898d-48aa-b4d9-87819af7098f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:22.986537 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:22.986376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087275 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/705e22ba-898d-48aa-b4d9-87819af7098f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087275 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087489 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/705e22ba-898d-48aa-b4d9-87819af7098f-kube-api-access-qmnbl\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087489 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/705e22ba-898d-48aa-b4d9-87819af7098f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087489 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.087489 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.087700 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:09:23.087562    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 13:09:23.087700 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:09:23.087636    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls podName:c651075f-f8e6-4183-b9cd-585f87e95cf8 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:23.587614479 +0000 UTC m=+1291.537960013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls") pod "isvc-canary-promote-predictor-8bc5b857-cwl9m" (UID: "c651075f-f8e6-4183-b9cd-585f87e95cf8") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 13:09:23.087700 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087700 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c651075f-f8e6-4183-b9cd-585f87e95cf8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.087944 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.087944 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/705e22ba-898d-48aa-b4d9-87819af7098f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.087944 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.087752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njx5k\" (UniqueName: \"kubernetes.io/projected/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-api-access-njx5k\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.088345 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.088320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.088461 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.088345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.088461 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.088377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c651075f-f8e6-4183-b9cd-585f87e95cf8-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.088461 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.088438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.088585 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.088499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.090030 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.090007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/705e22ba-898d-48aa-b4d9-87819af7098f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.096193 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.096171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njx5k\" (UniqueName: \"kubernetes.io/projected/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-api-access-njx5k\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.096472 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.096450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/705e22ba-898d-48aa-b4d9-87819af7098f-kube-api-access-qmnbl\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-hcx6g\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.172959 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.172904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:23.447159 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.447101    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"]
Sep 16 13:09:23.449534 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:09:23.449505    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705e22ba_898d_48aa_b4d9_87819af7098f.slice/crio-7b835c7e82b05a8007514727e9da2e14b0788e43a7010209ebe7406db21f5697 WatchSource:0}: Error finding container 7b835c7e82b05a8007514727e9da2e14b0788e43a7010209ebe7406db21f5697: Status 404 returned error can't find the container with id 7b835c7e82b05a8007514727e9da2e14b0788e43a7010209ebe7406db21f5697
Sep 16 13:09:23.592038 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.592010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.594340 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.594314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-cwl9m\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:23.741403 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.741336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerStarted","Data":"dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61"}
Sep 16 13:09:23.741403 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.741385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerStarted","Data":"7b835c7e82b05a8007514727e9da2e14b0788e43a7010209ebe7406db21f5697"}
Sep 16 13:09:23.780718 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:23.780690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:24.054956 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:24.054931    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"]
Sep 16 13:09:24.056927 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:09:24.056896    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc651075f_f8e6_4183_b9cd_585f87e95cf8.slice/crio-c3616458116afeed194291772779947bbc369267ae43c9895d06bca876f77c90 WatchSource:0}: Error finding container c3616458116afeed194291772779947bbc369267ae43c9895d06bca876f77c90: Status 404 returned error can't find the container with id c3616458116afeed194291772779947bbc369267ae43c9895d06bca876f77c90
Sep 16 13:09:24.748278 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:24.748210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerStarted","Data":"7689687085e38ed2609d722af48ea61710d14d4ac351a3c458ad4fcd4b182883"}
Sep 16 13:09:24.748493 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:24.748352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerStarted","Data":"c3616458116afeed194291772779947bbc369267ae43c9895d06bca876f77c90"}
Sep 16 13:09:27.763135 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:27.763100    2577 generic.go:356] "Generic (PLEG): container finished" podID="705e22ba-898d-48aa-b4d9-87819af7098f" containerID="dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61" exitCode=0
Sep 16 13:09:27.763471 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:27.763173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerDied","Data":"dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61"}
Sep 16 13:09:28.769152 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:28.769114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerStarted","Data":"c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76"}
Sep 16 13:09:28.769539 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:28.769190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerStarted","Data":"d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9"}
Sep 16 13:09:28.769539 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:28.769402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:28.770687 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:28.770664    2577 generic.go:356] "Generic (PLEG): container finished" podID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerID="7689687085e38ed2609d722af48ea61710d14d4ac351a3c458ad4fcd4b182883" exitCode=0
Sep 16 13:09:28.770812 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:28.770732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerDied","Data":"7689687085e38ed2609d722af48ea61710d14d4ac351a3c458ad4fcd4b182883"}
Sep 16 13:09:28.790807 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:28.790767    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podStartSLOduration=2.912129562 podStartE2EDuration="6.790752934s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878623372s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:28.788602833 +0000 UTC m=+1296.738948399" watchObservedRunningTime="2026-09-16 13:09:28.790752934 +0000 UTC m=+1296.741098487"
Sep 16 13:09:29.777420 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:29.777384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerStarted","Data":"8739d8e1d6fa444876543ba59f28ebd913cefa2afed178504b0148bc0c3e0c31"}
Sep 16 13:09:29.777939 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:29.777455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerStarted","Data":"bb7c43062f76f5917a37642e61ffa560ebb47ba0c0aa0dfe23ad072fb57f8bf3"}
Sep 16 13:09:29.777939 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:29.777538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:29.777939 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:29.777725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:29.779001 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:29.778976    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:09:29.796815 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:29.796774    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podStartSLOduration=3.923398961 podStartE2EDuration="7.796760687s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.873361726s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:29.795259678 +0000 UTC m=+1297.745605259" watchObservedRunningTime="2026-09-16 13:09:29.796760687 +0000 UTC m=+1297.747106240"
Sep 16 13:09:30.782362 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:30.782312    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:09:30.782914 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:30.782544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:30.783953 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:30.783931    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:09:31.793153 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:31.793112    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:09:35.787411 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:35.787386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:09:35.787938 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:35.787911    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:09:36.798593 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:36.798566    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:09:36.799226 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:36.799199    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:09:45.788339 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:45.788283    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:09:46.799257 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:46.799221    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:09:55.788677 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:55.788634    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:09:56.799980 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:09:56.799941    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:10:05.788231 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:05.788186    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:10:06.799085 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:06.799050    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:10:15.788147 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:15.788062    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:10:16.799256 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:16.799220    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:10:25.788540 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:25.788497    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:10:26.800059 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:26.800022    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:10:35.788051 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:35.788008    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:10:36.799390 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:36.799352    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 13:10:45.788984 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:45.788951    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:10:45.925718 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:45.925692    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"]
Sep 16 13:10:45.925986 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:45.925961    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container" containerID="cri-o://bb7c43062f76f5917a37642e61ffa560ebb47ba0c0aa0dfe23ad072fb57f8bf3" gracePeriod=30
Sep 16 13:10:45.926050 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:45.925979    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kube-rbac-proxy" containerID="cri-o://8739d8e1d6fa444876543ba59f28ebd913cefa2afed178504b0148bc0c3e0c31" gracePeriod=30
Sep 16 13:10:45.926800 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:45.926759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:10:46.145340 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:46.145308    2577 generic.go:356] "Generic (PLEG): container finished" podID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerID="8739d8e1d6fa444876543ba59f28ebd913cefa2afed178504b0148bc0c3e0c31" exitCode=2
Sep 16 13:10:46.145509 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:46.145356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerDied","Data":"8739d8e1d6fa444876543ba59f28ebd913cefa2afed178504b0148bc0c3e0c31"}
Sep 16 13:10:46.793392 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:46.793355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 13:10:50.166090 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.166054    2577 generic.go:356] "Generic (PLEG): container finished" podID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerID="bb7c43062f76f5917a37642e61ffa560ebb47ba0c0aa0dfe23ad072fb57f8bf3" exitCode=0
Sep 16 13:10:50.166380 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.166120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerDied","Data":"bb7c43062f76f5917a37642e61ffa560ebb47ba0c0aa0dfe23ad072fb57f8bf3"}
Sep 16 13:10:50.166380 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.166176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m" event={"ID":"c651075f-f8e6-4183-b9cd-585f87e95cf8","Type":"ContainerDied","Data":"c3616458116afeed194291772779947bbc369267ae43c9895d06bca876f77c90"}
Sep 16 13:10:50.166380 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.166205    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3616458116afeed194291772779947bbc369267ae43c9895d06bca876f77c90"
Sep 16 13:10:50.175383 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.175366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:10:50.247989 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.247966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-rbac-proxy-sar-config\") pod \"c651075f-f8e6-4183-b9cd-585f87e95cf8\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") "
Sep 16 13:10:50.248128 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls\") pod \"c651075f-f8e6-4183-b9cd-585f87e95cf8\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") "
Sep 16 13:10:50.248128 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njx5k\" (UniqueName: \"kubernetes.io/projected/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-api-access-njx5k\") pod \"c651075f-f8e6-4183-b9cd-585f87e95cf8\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") "
Sep 16 13:10:50.248128 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248083    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c651075f-f8e6-4183-b9cd-585f87e95cf8-kserve-provision-location\") pod \"c651075f-f8e6-4183-b9cd-585f87e95cf8\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") "
Sep 16 13:10:50.248299 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-cabundle-cert\") pod \"c651075f-f8e6-4183-b9cd-585f87e95cf8\" (UID: \"c651075f-f8e6-4183-b9cd-585f87e95cf8\") "
Sep 16 13:10:50.248444 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c651075f-f8e6-4183-b9cd-585f87e95cf8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c651075f-f8e6-4183-b9cd-585f87e95cf8" (UID: "c651075f-f8e6-4183-b9cd-585f87e95cf8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:10:50.248557 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c651075f-f8e6-4183-b9cd-585f87e95cf8" (UID: "c651075f-f8e6-4183-b9cd-585f87e95cf8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:10:50.248557 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.248518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c651075f-f8e6-4183-b9cd-585f87e95cf8" (UID: "c651075f-f8e6-4183-b9cd-585f87e95cf8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:10:50.250326 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.250299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c651075f-f8e6-4183-b9cd-585f87e95cf8" (UID: "c651075f-f8e6-4183-b9cd-585f87e95cf8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:10:50.250326 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.250311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-api-access-njx5k" (OuterVolumeSpecName: "kube-api-access-njx5k") pod "c651075f-f8e6-4183-b9cd-585f87e95cf8" (UID: "c651075f-f8e6-4183-b9cd-585f87e95cf8"). InnerVolumeSpecName "kube-api-access-njx5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:10:50.349522 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.349489    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c651075f-f8e6-4183-b9cd-585f87e95cf8-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:10:50.349522 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.349519    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:10:50.349522 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.349528    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:10:50.349693 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.349537    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c651075f-f8e6-4183-b9cd-585f87e95cf8-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:10:50.349693 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:50.349546    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njx5k\" (UniqueName: \"kubernetes.io/projected/c651075f-f8e6-4183-b9cd-585f87e95cf8-kube-api-access-njx5k\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:10:51.170654 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:51.170622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"
Sep 16 13:10:51.189211 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:51.189154    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"]
Sep 16 13:10:51.192931 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:51.192908    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-cwl9m"]
Sep 16 13:10:52.573599 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:10:52.573557    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" path="/var/lib/kubelet/pods/c651075f-f8e6-4183-b9cd-585f87e95cf8/volumes"
Sep 16 13:12:23.150678 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:23.150640    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"]
Sep 16 13:12:23.151154 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:23.150897    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" containerID="cri-o://d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9" gracePeriod=30
Sep 16 13:12:23.151154 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:23.150947    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kube-rbac-proxy" containerID="cri-o://c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76" gracePeriod=30
Sep 16 13:12:23.599557 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:23.599525    2577 generic.go:356] "Generic (PLEG): container finished" podID="705e22ba-898d-48aa-b4d9-87819af7098f" containerID="c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76" exitCode=2
Sep 16 13:12:23.599721 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:23.599585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerDied","Data":"c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76"}
Sep 16 13:12:25.782979 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:25.782937    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 16 13:12:25.788340 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:25.788317    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 13:12:27.597605 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.597581    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:12:27.619544 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.619518    2577 generic.go:356] "Generic (PLEG): container finished" podID="705e22ba-898d-48aa-b4d9-87819af7098f" containerID="d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9" exitCode=0
Sep 16 13:12:27.619651 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.619587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerDied","Data":"d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9"}
Sep 16 13:12:27.619651 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.619632    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"
Sep 16 13:12:27.619735 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.619676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g" event={"ID":"705e22ba-898d-48aa-b4d9-87819af7098f","Type":"ContainerDied","Data":"7b835c7e82b05a8007514727e9da2e14b0788e43a7010209ebe7406db21f5697"}
Sep 16 13:12:27.619735 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.619717    2577 scope.go:117] "RemoveContainer" containerID="c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76"
Sep 16 13:12:27.628130 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.628112    2577 scope.go:117] "RemoveContainer" containerID="d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9"
Sep 16 13:12:27.635755 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.635735    2577 scope.go:117] "RemoveContainer" containerID="dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61"
Sep 16 13:12:27.643293 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.643276    2577 scope.go:117] "RemoveContainer" containerID="c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76"
Sep 16 13:12:27.643567 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:12:27.643550    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76\": container with ID starting with c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76 not found: ID does not exist" containerID="c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76"
Sep 16 13:12:27.643640 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.643578    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76"} err="failed to get container status \"c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76\": rpc error: code = NotFound desc = could not find container \"c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76\": container with ID starting with c12a37df6581ae1e3e97c8693db761c43680d6c26546193b8bdc60f19ebf1e76 not found: ID does not exist"
Sep 16 13:12:27.643640 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.643599    2577 scope.go:117] "RemoveContainer" containerID="d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9"
Sep 16 13:12:27.643844 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:12:27.643827    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9\": container with ID starting with d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9 not found: ID does not exist" containerID="d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9"
Sep 16 13:12:27.643927 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.643851    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9"} err="failed to get container status \"d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9\": rpc error: code = NotFound desc = could not find container \"d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9\": container with ID starting with d68c54e5acba8f3f5406de79e768abf44c422e7592b286862243392062119ce9 not found: ID does not exist"
Sep 16 13:12:27.643927 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.643887    2577 scope.go:117] "RemoveContainer" containerID="dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61"
Sep 16 13:12:27.644140 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:12:27.644123    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61\": container with ID starting with dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61 not found: ID does not exist" containerID="dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61"
Sep 16 13:12:27.644181 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.644144    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61"} err="failed to get container status \"dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61\": rpc error: code = NotFound desc = could not find container \"dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61\": container with ID starting with dd82751f9610ac0e97cee6adc6f67e35bc241915c308635a76255a2250d83b61 not found: ID does not exist"
Sep 16 13:12:27.713146 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/705e22ba-898d-48aa-b4d9-87819af7098f-kube-api-access-qmnbl\") pod \"705e22ba-898d-48aa-b4d9-87819af7098f\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") "
Sep 16 13:12:27.713146 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/705e22ba-898d-48aa-b4d9-87819af7098f-proxy-tls\") pod \"705e22ba-898d-48aa-b4d9-87819af7098f\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") "
Sep 16 13:12:27.713299 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713176    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-kube-rbac-proxy-sar-config\") pod \"705e22ba-898d-48aa-b4d9-87819af7098f\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") "
Sep 16 13:12:27.713299 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-cabundle-cert\") pod \"705e22ba-898d-48aa-b4d9-87819af7098f\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") "
Sep 16 13:12:27.713299 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/705e22ba-898d-48aa-b4d9-87819af7098f-kserve-provision-location\") pod \"705e22ba-898d-48aa-b4d9-87819af7098f\" (UID: \"705e22ba-898d-48aa-b4d9-87819af7098f\") "
Sep 16 13:12:27.713594 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "705e22ba-898d-48aa-b4d9-87819af7098f" (UID: "705e22ba-898d-48aa-b4d9-87819af7098f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:12:27.713686 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/705e22ba-898d-48aa-b4d9-87819af7098f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "705e22ba-898d-48aa-b4d9-87819af7098f" (UID: "705e22ba-898d-48aa-b4d9-87819af7098f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:27.713686 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.713572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "705e22ba-898d-48aa-b4d9-87819af7098f" (UID: "705e22ba-898d-48aa-b4d9-87819af7098f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:12:27.715209 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.715185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705e22ba-898d-48aa-b4d9-87819af7098f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "705e22ba-898d-48aa-b4d9-87819af7098f" (UID: "705e22ba-898d-48aa-b4d9-87819af7098f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:27.715322 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.715230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705e22ba-898d-48aa-b4d9-87819af7098f-kube-api-access-qmnbl" (OuterVolumeSpecName: "kube-api-access-qmnbl") pod "705e22ba-898d-48aa-b4d9-87819af7098f" (UID: "705e22ba-898d-48aa-b4d9-87819af7098f"). InnerVolumeSpecName "kube-api-access-qmnbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:27.814171 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.814145    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/705e22ba-898d-48aa-b4d9-87819af7098f-kserve-provision-location\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.814171 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.814167    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnbl\" (UniqueName: \"kubernetes.io/projected/705e22ba-898d-48aa-b4d9-87819af7098f-kube-api-access-qmnbl\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.814307 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.814176    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/705e22ba-898d-48aa-b4d9-87819af7098f-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.814307 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.814186    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.814307 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.814194    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/705e22ba-898d-48aa-b4d9-87819af7098f-cabundle-cert\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:12:27.941650 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.941626    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"]
Sep 16 13:12:27.945961 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:27.945938    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-hcx6g"]
Sep 16 13:12:28.573218 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:28.573187    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" path="/var/lib/kubelet/pods/705e22ba-898d-48aa-b4d9-87819af7098f/volumes"
Sep 16 13:12:37.047729 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.047696    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"]
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container"
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048214    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container"
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048236    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="storage-initializer"
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048246    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="storage-initializer"
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048260    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container"
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048268    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container"
Sep 16 13:12:37.048287 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048289    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="storage-initializer"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048313    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="storage-initializer"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048329    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kube-rbac-proxy"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kube-rbac-proxy"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048348    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kube-rbac-proxy"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048357    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kube-rbac-proxy"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048483    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kube-rbac-proxy"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048502    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kube-rbac-proxy"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048513    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="705e22ba-898d-48aa-b4d9-87819af7098f" containerName="kserve-container"
Sep 16 13:12:37.048578 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.048522    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c651075f-f8e6-4183-b9cd-585f87e95cf8" containerName="kserve-container"
Sep 16 13:12:37.053096 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.053080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.055820 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.055800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9sk5b\""
Sep 16 13:12:37.055971 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.055844    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 13:12:37.056623 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.056607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 13:12:37.056824 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.056809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 13:12:37.056959 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.056941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 16 13:12:37.061007 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.060988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"]
Sep 16 13:12:37.188955 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.188929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30b9716-d5bc-4c88-94b2-d877b8bf6708-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.189068 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.188963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.189068 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.188980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzpt\" (UniqueName: \"kubernetes.io/projected/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-api-access-qxzpt\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.289997 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.289972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30b9716-d5bc-4c88-94b2-d877b8bf6708-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.290114 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.290002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.290114 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.290017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzpt\" (UniqueName: \"kubernetes.io/projected/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-api-access-qxzpt\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.290635 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.290616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.292538 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.292517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30b9716-d5bc-4c88-94b2-d877b8bf6708-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.297897 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.297827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzpt\" (UniqueName: \"kubernetes.io/projected/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-api-access-qxzpt\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bstz5\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.364000 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.363976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:37.632912 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.632819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"]
Sep 16 13:12:37.635716 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:12:37.635682    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30b9716_d5bc_4c88_94b2_d877b8bf6708.slice/crio-b8a310397d4b219fa186310ad9d53275e8c0d3f036e3af35117dc5f23d8a4654 WatchSource:0}: Error finding container b8a310397d4b219fa186310ad9d53275e8c0d3f036e3af35117dc5f23d8a4654: Status 404 returned error can't find the container with id b8a310397d4b219fa186310ad9d53275e8c0d3f036e3af35117dc5f23d8a4654
Sep 16 13:12:37.668464 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:37.668440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" event={"ID":"c30b9716-d5bc-4c88-94b2-d877b8bf6708","Type":"ContainerStarted","Data":"b8a310397d4b219fa186310ad9d53275e8c0d3f036e3af35117dc5f23d8a4654"}
Sep 16 13:12:38.674840 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:38.674797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" event={"ID":"c30b9716-d5bc-4c88-94b2-d877b8bf6708","Type":"ContainerStarted","Data":"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"}
Sep 16 13:12:38.675212 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:38.674898    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:38.675212 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:38.674933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:38.675212 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:38.674960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" event={"ID":"c30b9716-d5bc-4c88-94b2-d877b8bf6708","Type":"ContainerStarted","Data":"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"}
Sep 16 13:12:38.676533 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:38.676512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:38.692669 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:38.692626    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" podStartSLOduration=1.692611926 podStartE2EDuration="1.692611926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:38.690368397 +0000 UTC m=+1486.640713951" watchObservedRunningTime="2026-09-16 13:12:38.692611926 +0000 UTC m=+1486.642957479"
Sep 16 13:12:45.688240 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:45.688211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:12:52.614078 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:52.614053    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:12:52.617319 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:12:52.615469    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:14:22.204068 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.204035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-bstz5_c30b9716-d5bc-4c88-94b2-d877b8bf6708/kserve-container/0.log"
Sep 16 13:14:22.431601 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.431568    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"]
Sep 16 13:14:22.431852 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.431800    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kserve-container" containerID="cri-o://7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032" gracePeriod=30
Sep 16 13:14:22.431852 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.431821    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kube-rbac-proxy" containerID="cri-o://2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739" gracePeriod=30
Sep 16 13:14:22.682585 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.682562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:14:22.713336 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.713310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-rbac-proxy-sar-config\") pod \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") "
Sep 16 13:14:22.713489 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.713420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxzpt\" (UniqueName: \"kubernetes.io/projected/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-api-access-qxzpt\") pod \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") "
Sep 16 13:14:22.713489 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.713458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30b9716-d5bc-4c88-94b2-d877b8bf6708-proxy-tls\") pod \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\" (UID: \"c30b9716-d5bc-4c88-94b2-d877b8bf6708\") "
Sep 16 13:14:22.713695 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.713669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c30b9716-d5bc-4c88-94b2-d877b8bf6708" (UID: "c30b9716-d5bc-4c88-94b2-d877b8bf6708"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:14:22.715508 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.715481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-api-access-qxzpt" (OuterVolumeSpecName: "kube-api-access-qxzpt") pod "c30b9716-d5bc-4c88-94b2-d877b8bf6708" (UID: "c30b9716-d5bc-4c88-94b2-d877b8bf6708"). InnerVolumeSpecName "kube-api-access-qxzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:22.715508 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.715495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30b9716-d5bc-4c88-94b2-d877b8bf6708-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c30b9716-d5bc-4c88-94b2-d877b8bf6708" (UID: "c30b9716-d5bc-4c88-94b2-d877b8bf6708"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:22.815147 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.815084    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxzpt\" (UniqueName: \"kubernetes.io/projected/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-api-access-qxzpt\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:14:22.815147 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.815108    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30b9716-d5bc-4c88-94b2-d877b8bf6708-proxy-tls\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:14:22.815147 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:22.815120    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30b9716-d5bc-4c88-94b2-d877b8bf6708-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-99.ec2.internal\" DevicePath \"\""
Sep 16 13:14:23.159903 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.159804    2577 generic.go:356] "Generic (PLEG): container finished" podID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerID="2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739" exitCode=2
Sep 16 13:14:23.159903 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.159832    2577 generic.go:356] "Generic (PLEG): container finished" podID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerID="7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032" exitCode=2
Sep 16 13:14:23.159903 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.159850    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"
Sep 16 13:14:23.160106 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.159899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" event={"ID":"c30b9716-d5bc-4c88-94b2-d877b8bf6708","Type":"ContainerDied","Data":"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"}
Sep 16 13:14:23.160106 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.159967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" event={"ID":"c30b9716-d5bc-4c88-94b2-d877b8bf6708","Type":"ContainerDied","Data":"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"}
Sep 16 13:14:23.160106 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.160003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5" event={"ID":"c30b9716-d5bc-4c88-94b2-d877b8bf6708","Type":"ContainerDied","Data":"b8a310397d4b219fa186310ad9d53275e8c0d3f036e3af35117dc5f23d8a4654"}
Sep 16 13:14:23.160106 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.160039    2577 scope.go:117] "RemoveContainer" containerID="2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"
Sep 16 13:14:23.169241 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.169223    2577 scope.go:117] "RemoveContainer" containerID="7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"
Sep 16 13:14:23.177132 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177113    2577 scope.go:117] "RemoveContainer" containerID="2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"
Sep 16 13:14:23.177400 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:14:23.177383    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739\": container with ID starting with 2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739 not found: ID does not exist" containerID="2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"
Sep 16 13:14:23.177450 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177410    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"} err="failed to get container status \"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739\": rpc error: code = NotFound desc = could not find container \"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739\": container with ID starting with 2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739 not found: ID does not exist"
Sep 16 13:14:23.177450 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177425    2577 scope.go:117] "RemoveContainer" containerID="7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"
Sep 16 13:14:23.177684 ip-10-0-143-99 kubenswrapper[2577]: E0916 13:14:23.177668    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032\": container with ID starting with 7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032 not found: ID does not exist" containerID="7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"
Sep 16 13:14:23.177736 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177688    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"} err="failed to get container status \"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032\": rpc error: code = NotFound desc = could not find container \"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032\": container with ID starting with 7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032 not found: ID does not exist"
Sep 16 13:14:23.177736 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177702    2577 scope.go:117] "RemoveContainer" containerID="2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"
Sep 16 13:14:23.177963 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177943    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739"} err="failed to get container status \"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739\": rpc error: code = NotFound desc = could not find container \"2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739\": container with ID starting with 2ae44fced38c6bd97a4137643cabfec5b2144677686fc6f9c0ba2ac9747ec739 not found: ID does not exist"
Sep 16 13:14:23.178052 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.177966    2577 scope.go:117] "RemoveContainer" containerID="7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"
Sep 16 13:14:23.178234 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.178210    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032"} err="failed to get container status \"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032\": rpc error: code = NotFound desc = could not find container \"7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032\": container with ID starting with 7abc396af4783c96ef44c8230ea8266358dd586bbcb5752de53e23f8a95db032 not found: ID does not exist"
Sep 16 13:14:23.181815 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.181790    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"]
Sep 16 13:14:23.185229 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:23.185209    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bstz5"]
Sep 16 13:14:24.573001 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:24.572967    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" path="/var/lib/kubelet/pods/c30b9716-d5bc-4c88-94b2-d877b8bf6708/volumes"
Sep 16 13:14:55.325825 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.325791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l458r/must-gather-rx7hs"]
Sep 16 13:14:55.326290 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.326163    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kube-rbac-proxy"
Sep 16 13:14:55.326290 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.326177    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kube-rbac-proxy"
Sep 16 13:14:55.326290 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.326204    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kserve-container"
Sep 16 13:14:55.326290 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.326209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kserve-container"
Sep 16 13:14:55.326290 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.326267    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kube-rbac-proxy"
Sep 16 13:14:55.326290 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.326276    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30b9716-d5bc-4c88-94b2-d877b8bf6708" containerName="kserve-container"
Sep 16 13:14:55.328866 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.328837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.331305 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.331271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l458r\"/\"kube-root-ca.crt\""
Sep 16 13:14:55.331441 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.331324    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l458r\"/\"openshift-service-ca.crt\""
Sep 16 13:14:55.332420 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.332393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l458r\"/\"default-dockercfg-rmglt\""
Sep 16 13:14:55.336596 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.336574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l458r/must-gather-rx7hs"]
Sep 16 13:14:55.478366 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.478309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6523145-468c-44c9-9488-5dfb8983ed40-must-gather-output\") pod \"must-gather-rx7hs\" (UID: \"a6523145-468c-44c9-9488-5dfb8983ed40\") " pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.478574 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.478477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gp9z\" (UniqueName: \"kubernetes.io/projected/a6523145-468c-44c9-9488-5dfb8983ed40-kube-api-access-8gp9z\") pod \"must-gather-rx7hs\" (UID: \"a6523145-468c-44c9-9488-5dfb8983ed40\") " pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.579822 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.579725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6523145-468c-44c9-9488-5dfb8983ed40-must-gather-output\") pod \"must-gather-rx7hs\" (UID: \"a6523145-468c-44c9-9488-5dfb8983ed40\") " pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.580010 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.579844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gp9z\" (UniqueName: \"kubernetes.io/projected/a6523145-468c-44c9-9488-5dfb8983ed40-kube-api-access-8gp9z\") pod \"must-gather-rx7hs\" (UID: \"a6523145-468c-44c9-9488-5dfb8983ed40\") " pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.580090 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.580071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6523145-468c-44c9-9488-5dfb8983ed40-must-gather-output\") pod \"must-gather-rx7hs\" (UID: \"a6523145-468c-44c9-9488-5dfb8983ed40\") " pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.588457 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.588427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gp9z\" (UniqueName: \"kubernetes.io/projected/a6523145-468c-44c9-9488-5dfb8983ed40-kube-api-access-8gp9z\") pod \"must-gather-rx7hs\" (UID: \"a6523145-468c-44c9-9488-5dfb8983ed40\") " pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.655912 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.655887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l458r/must-gather-rx7hs"
Sep 16 13:14:55.926050 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.926021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l458r/must-gather-rx7hs"]
Sep 16 13:14:55.928809 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:14:55.928782    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6523145_468c_44c9_9488_5dfb8983ed40.slice/crio-e2d9e553b1911980f0b7a9f66f4f95652412daa4f6021ab552c6a526092d8d41 WatchSource:0}: Error finding container e2d9e553b1911980f0b7a9f66f4f95652412daa4f6021ab552c6a526092d8d41: Status 404 returned error can't find the container with id e2d9e553b1911980f0b7a9f66f4f95652412daa4f6021ab552c6a526092d8d41
Sep 16 13:14:55.931020 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:55.930996    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:14:56.310253 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:56.310220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/must-gather-rx7hs" event={"ID":"a6523145-468c-44c9-9488-5dfb8983ed40","Type":"ContainerStarted","Data":"e2d9e553b1911980f0b7a9f66f4f95652412daa4f6021ab552c6a526092d8d41"}
Sep 16 13:14:57.320182 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:57.319374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/must-gather-rx7hs" event={"ID":"a6523145-468c-44c9-9488-5dfb8983ed40","Type":"ContainerStarted","Data":"5b9c03ed417012acc3d972b9c8cd3ce42b8ecddc23fe555a27a6a0085d5480a3"}
Sep 16 13:14:57.320182 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:57.319426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/must-gather-rx7hs" event={"ID":"a6523145-468c-44c9-9488-5dfb8983ed40","Type":"ContainerStarted","Data":"3cb40d76784ce5d38cb349b01b0a1be183e0c0c9402fad9568b1dcc2599a3692"}
Sep 16 13:14:57.337218 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:57.337175    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l458r/must-gather-rx7hs" podStartSLOduration=1.432885277 podStartE2EDuration="2.337160045s" totalImagesPullingTime="904.274768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:57.334984542 +0000 UTC m=+1625.285330096" watchObservedRunningTime="2026-09-16 13:14:57.337160045 +0000 UTC m=+1625.287505597"
Sep 16 13:14:58.169145 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:58.169112    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bj87h_15cbea84-3f63-497b-b7db-7e88193b54d6/global-pull-secret-syncer/0.log"
Sep 16 13:14:58.288477 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:58.288434    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xnvx4_588f3d87-3ab5-4b39-bb0b-2636e8411bb9/konnectivity-agent/0.log"
Sep 16 13:14:58.346462 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:14:58.346436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-99.ec2.internal_562cd238c1a07a23034785db4cb30206/haproxy/0.log"
Sep 16 13:15:02.293464 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.293432    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/alertmanager/0.log"
Sep 16 13:15:02.314982 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.314938    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/config-reloader/0.log"
Sep 16 13:15:02.337488 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.337418    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/kube-rbac-proxy-web/0.log"
Sep 16 13:15:02.358507 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.358478    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/kube-rbac-proxy/0.log"
Sep 16 13:15:02.376567 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.376527    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/kube-rbac-proxy-metric/0.log"
Sep 16 13:15:02.400240 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.400215    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/prom-label-proxy/0.log"
Sep 16 13:15:02.422903 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.422852    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba3b4bde-d865-4638-8e83-21fd26085131/init-config-reloader/0.log"
Sep 16 13:15:02.474581 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.474556    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-sfhlf_4f68d64d-28fc-411d-85ec-8fc331c3e756/cluster-monitoring-operator/0.log"
Sep 16 13:15:02.588213 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.588134    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75f78dbf54-4bw5b_ac2ce0be-e36a-4f99-9884-b5d6a233796c/metrics-server/0.log"
Sep 16 13:15:02.637232 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.637202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-l7sl9_776bc35e-d960-4fe2-a6d0-5ab3c4f08c4d/monitoring-plugin/0.log"
Sep 16 13:15:02.839305 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.839232    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzkpg_72e638f8-6925-4822-a940-b9318f0d13bd/node-exporter/0.log"
Sep 16 13:15:02.858818 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.858787    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzkpg_72e638f8-6925-4822-a940-b9318f0d13bd/kube-rbac-proxy/0.log"
Sep 16 13:15:02.880617 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:02.880583    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tzkpg_72e638f8-6925-4822-a940-b9318f0d13bd/init-textfile/0.log"
Sep 16 13:15:03.144279 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.144196    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kshmf_43cabd51-e2f9-4770-8228-0a83e785cd78/prometheus-operator/0.log"
Sep 16 13:15:03.160733 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.160674    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kshmf_43cabd51-e2f9-4770-8228-0a83e785cd78/kube-rbac-proxy/0.log"
Sep 16 13:15:03.209672 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.209623    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69b68dc588-p9mgt_216f4d9d-2aad-43a1-aa3a-4fe6ba83a096/telemeter-client/0.log"
Sep 16 13:15:03.227900 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.227845    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69b68dc588-p9mgt_216f4d9d-2aad-43a1-aa3a-4fe6ba83a096/reload/0.log"
Sep 16 13:15:03.249759 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.249735    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69b68dc588-p9mgt_216f4d9d-2aad-43a1-aa3a-4fe6ba83a096/kube-rbac-proxy/0.log"
Sep 16 13:15:03.283278 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.283244    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdfd595d9-thbtv_0f57d20f-7e64-4959-aa61-61537eb7bd55/thanos-query/0.log"
Sep 16 13:15:03.306692 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.306665    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdfd595d9-thbtv_0f57d20f-7e64-4959-aa61-61537eb7bd55/kube-rbac-proxy-web/0.log"
Sep 16 13:15:03.326373 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.326350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdfd595d9-thbtv_0f57d20f-7e64-4959-aa61-61537eb7bd55/kube-rbac-proxy/0.log"
Sep 16 13:15:03.347935 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.347908    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdfd595d9-thbtv_0f57d20f-7e64-4959-aa61-61537eb7bd55/prom-label-proxy/0.log"
Sep 16 13:15:03.373789 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.373734    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdfd595d9-thbtv_0f57d20f-7e64-4959-aa61-61537eb7bd55/kube-rbac-proxy-rules/0.log"
Sep 16 13:15:03.392280 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:03.392248    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bdfd595d9-thbtv_0f57d20f-7e64-4959-aa61-61537eb7bd55/kube-rbac-proxy-metrics/0.log"
Sep 16 13:15:05.228357 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:05.228321    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/1.log"
Sep 16 13:15:05.235686 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:05.235663    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mbvzm_088a59c6-e72f-409f-b775-369532584ed0/console-operator/2.log"
Sep 16 13:15:05.650062 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:05.650033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-595b95f48b-5tqfb_fb47497e-18ab-43f1-bc6c-f098ea1c46a5/console/0.log"
Sep 16 13:15:05.678167 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:05.678142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-pddqc_ce9d7c0c-ba72-40ae-a0a6-6c2009ff74aa/download-server/0.log"
Sep 16 13:15:07.135884 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.135841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9s7f_78064f5c-911d-470b-9635-eafb7e7ee14c/dns/0.log"
Sep 16 13:15:07.142135 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.142106    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"]
Sep 16 13:15:07.148550 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.148524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.154929 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.154901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"]
Sep 16 13:15:07.161250 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.161226    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9s7f_78064f5c-911d-470b-9635-eafb7e7ee14c/kube-rbac-proxy/0.log"
Sep 16 13:15:07.206735 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.206705    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p6wfm_11fb8233-0ed8-4d69-ba5f-0d998d7f2ca5/dns-node-resolver/0.log"
Sep 16 13:15:07.208399 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.208374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-sys\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.208546 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.208407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ft2\" (UniqueName: \"kubernetes.io/projected/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-kube-api-access-59ft2\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.208546 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.208441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-podres\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.208546 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.208521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-proc\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.208653 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.208615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-lib-modules\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-proc\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-lib-modules\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-sys\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ft2\" (UniqueName: \"kubernetes.io/projected/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-kube-api-access-59ft2\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-podres\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-podres\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-proc\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-lib-modules\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.309930 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.309684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-sys\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.318551 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.318525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ft2\" (UniqueName: \"kubernetes.io/projected/f18ecf5d-e389-4d94-92ab-b8e1ae1080f7-kube-api-access-59ft2\") pod \"perf-node-gather-daemonset-tzt6s\" (UID: \"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7\") " pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.460766 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.460738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:07.696704 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.696670    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55685f76cf-92pmk_cb285905-617d-4e32-870a-9ba9b7c6380d/registry/0.log"
Sep 16 13:15:07.747907 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.747884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"]
Sep 16 13:15:07.750167 ip-10-0-143-99 kubenswrapper[2577]: W0916 13:15:07.750133    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf18ecf5d_e389_4d94_92ab_b8e1ae1080f7.slice/crio-cd11533028158e43b9aa499f039004274f3d64aa860fd3d3f27a369469e171ad WatchSource:0}: Error finding container cd11533028158e43b9aa499f039004274f3d64aa860fd3d3f27a369469e171ad: Status 404 returned error can't find the container with id cd11533028158e43b9aa499f039004274f3d64aa860fd3d3f27a369469e171ad
Sep 16 13:15:07.762372 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:07.762343    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jkzg_5759cfbb-d3e5-4129-b62e-00648ab43d8b/node-ca/0.log"
Sep 16 13:15:08.405080 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:08.405028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s" event={"ID":"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7","Type":"ContainerStarted","Data":"0763869ddaafa92f11c39f14d1c0a312455b4f9f10319361169a9ac20086399e"}
Sep 16 13:15:08.405517 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:08.405156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s" event={"ID":"f18ecf5d-e389-4d94-92ab-b8e1ae1080f7","Type":"ContainerStarted","Data":"cd11533028158e43b9aa499f039004274f3d64aa860fd3d3f27a369469e171ad"}
Sep 16 13:15:08.405517 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:08.405352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:08.420978 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:08.420928    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s" podStartSLOduration=1.420912263 podStartE2EDuration="1.420912263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:08.420610809 +0000 UTC m=+1636.370956363" watchObservedRunningTime="2026-09-16 13:15:08.420912263 +0000 UTC m=+1636.371257837"
Sep 16 13:15:08.881889 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:08.881851    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9xxt_e825c3a2-5e2f-47be-8a80-0b0c74456208/serve-healthcheck-canary/0.log"
Sep 16 13:15:09.204078 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:09.204053    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-jgggh_4ba4cd64-504f-4bae-9f8b-be8a24205fca/insights-operator/0.log"
Sep 16 13:15:09.204769 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:09.204750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-jgggh_4ba4cd64-504f-4bae-9f8b-be8a24205fca/insights-operator/1.log"
Sep 16 13:15:09.339851 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:09.339813    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tjwqw_381d5120-3a5b-43b1-af75-56abb87e27b5/kube-rbac-proxy/0.log"
Sep 16 13:15:09.356884 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:09.356848    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tjwqw_381d5120-3a5b-43b1-af75-56abb87e27b5/exporter/0.log"
Sep 16 13:15:09.373834 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:09.373811    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tjwqw_381d5120-3a5b-43b1-af75-56abb87e27b5/extractor/0.log"
Sep 16 13:15:11.591442 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:11.591409    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-867644d759-qdzjt_ad550311-c5c2-4173-8ae2-4875cbdadd1d/manager/0.log"
Sep 16 13:15:11.633772 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:11.633745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-v94xv_056515a3-501a-4f54-b8c9-2b5a3b166396/server/0.log"
Sep 16 13:15:14.421146 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:14.421116    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l458r/perf-node-gather-daemonset-tzt6s"
Sep 16 13:15:15.947235 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:15.947205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-gj6m9_bef9cefd-33b8-413a-b33f-e158ac86980e/kube-storage-version-migrator-operator/1.log"
Sep 16 13:15:15.948116 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:15.948092    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-gj6m9_bef9cefd-33b8-413a-b33f-e158ac86980e/kube-storage-version-migrator-operator/0.log"
Sep 16 13:15:17.322391 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.322363    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/kube-multus-additional-cni-plugins/0.log"
Sep 16 13:15:17.342026 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.342003    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/egress-router-binary-copy/0.log"
Sep 16 13:15:17.362084 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.362054    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/cni-plugins/0.log"
Sep 16 13:15:17.380019 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.380000    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/bond-cni-plugin/0.log"
Sep 16 13:15:17.400143 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.400119    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/routeoverride-cni/0.log"
Sep 16 13:15:17.419097 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.419076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/whereabouts-cni-bincopy/0.log"
Sep 16 13:15:17.439750 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.439725    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmcvr_ff789be7-69c9-40bb-a8aa-42f34de68c8f/whereabouts-cni/0.log"
Sep 16 13:15:17.615443 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.615375    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jt5qw_9ce327f0-b212-4e77-a282-cd4e727bdc04/kube-multus/0.log"
Sep 16 13:15:17.742402 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.742371    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mww2q_8bb342f5-ba85-4bea-a1a5-fa75f526c896/network-metrics-daemon/0.log"
Sep 16 13:15:17.768596 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:17.768566    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mww2q_8bb342f5-ba85-4bea-a1a5-fa75f526c896/kube-rbac-proxy/0.log"
Sep 16 13:15:18.707162 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.707134    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/ovn-controller/0.log"
Sep 16 13:15:18.723750 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.723720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/ovn-acl-logging/0.log"
Sep 16 13:15:18.741993 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.741973    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/kube-rbac-proxy-node/0.log"
Sep 16 13:15:18.758144 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.758127    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 13:15:18.773517 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.773497    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/northd/0.log"
Sep 16 13:15:18.790827 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.790807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/nbdb/0.log"
Sep 16 13:15:18.809510 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.809490    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/sbdb/0.log"
Sep 16 13:15:18.848753 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:18.848733    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8psw_d9129bf1-95b9-47aa-83a7-2a76f3729b55/ovnkube-controller/0.log"
Sep 16 13:15:20.065802 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:20.065771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fr8w4_4ffbc9a0-7600-4bcb-be04-63eab62a6cad/network-check-target-container/0.log"
Sep 16 13:15:20.949570 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:20.949544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4h482_06a8f2ab-0bc2-4dc7-b4bd-1acc18095840/iptables-alerter/0.log"
Sep 16 13:15:21.633227 ip-10-0-143-99 kubenswrapper[2577]: I0916 13:15:21.633202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-q46rc_3bcbcabb-f2a8-4ed2-b6d0-a1d58c384f4e/tuned/0.log"