Sep 05 02:15:56.821466 ip-10-0-138-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 05 02:15:56.821826 ip-10-0-138-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 05 02:15:56.821952 ip-10-0-138-105 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 05 02:15:56.822477 ip-10-0-138-105 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 05 02:16:06.838554 ip-10-0-138-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 05 02:16:06.838569 ip-10-0-138-105 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7755f76ed0c7421f9815a5f18875740d --
Sep 05 02:17:38.369643 ip-10-0-138-105 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:17:38.795171 ip-10-0-138-105 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:38.795171 ip-10-0-138-105 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:17:38.795171 ip-10-0-138-105 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:38.795171 ip-10-0-138-105 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:17:38.795171 ip-10-0-138-105 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:38.796171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.796079    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:17:38.804596 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804572    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:38.804596 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804591    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:38.804596 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804596    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:38.804596 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804600    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:38.804596 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804603    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:38.804596 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804606    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804609    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804612    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804615    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804617    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804620    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804623    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804626    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804629    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804631    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804634    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804636    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804639    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804642    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804644    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804647    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804650    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804652    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804655    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804658    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:38.804843 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804660    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804663    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804666    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804668    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804671    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804674    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804676    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804679    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804682    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804684    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804688    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804693    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804696    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804698    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804702    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804704    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804707    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804710    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804713    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:38.805336 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804716    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804719    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804722    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804725    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804727    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804730    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804733    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804736    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804739    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804741    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804744    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804746    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804749    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804752    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804754    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804757    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804760    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804763    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804765    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:38.805810 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804768    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804770    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804773    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804775    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804779    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804782    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804784    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804787    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804790    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804793    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804796    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804798    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804801    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804804    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804806    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804809    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804812    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804815    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804817    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804822    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:38.806279 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804826    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804830    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804835    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804838    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804840    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804844    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804847    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804850    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804853    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804856    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804859    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804861    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804865    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804868    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804871    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.804874    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805587    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805594    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805597    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:38.806762 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805600    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805604    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805607    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805610    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805613    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805615    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805618    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805621    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805624    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805627    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805629    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805632    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805634    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805637    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805640    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805643    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805645    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805648    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805652    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:38.807226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805657    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805660    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805663    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805666    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805669    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805672    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805675    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805678    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805681    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805683    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805686    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805688    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805691    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805694    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805698    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805702    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805705    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805708    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805711    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805714    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:38.807738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805717    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805720    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805723    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805727    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805729    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805732    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805735    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805737    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805741    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805743    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805746    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805748    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805751    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805754    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805764    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805767    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805770    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805773    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805776    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805779    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:38.808233 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805782    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805785    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805787    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805791    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805794    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805797    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805799    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805802    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805805    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805807    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805810    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805813    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805817    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805819    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805823    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805826    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805829    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805832    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805834    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805837    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:38.808745 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805839    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805842    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805845    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805848    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805850    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805853    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805856    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805858    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805861    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805864    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805867    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805870    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805873    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805876    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805878    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805881    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.805883    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806319    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806325    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806329    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:38.809230 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806332    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806335    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806337    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806340    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806343    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806346    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806348    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806351    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806354    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806356    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806359    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806362    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806364    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806368    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806371    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806374    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806378    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806382    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806385    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:38.809729 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806388    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806390    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806393    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806396    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806399    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806402    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806404    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806407    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806410    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806412    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806415    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806418    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806420    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806423    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806426    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806428    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806431    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806433    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806436    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806439    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:38.810199 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806441    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806444    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806447    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806449    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806452    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806455    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806458    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806461    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806466    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806469    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806472    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806474    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806477    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806480    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806484    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806487    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806490    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806493    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806496    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806499    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:38.810682 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806501    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806505    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806507    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806510    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806513    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806516    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806518    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806521    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806524    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806527    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806529    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806532    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806534    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806539    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806541    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806544    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806546    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806549    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806552    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806555    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:38.811180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806561    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806565    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806568    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806572    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806575    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806577    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806581    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806583    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806586    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806588    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806591    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806594    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806596    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806599    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806601    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806604    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.806607    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806678    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806717    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806724    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:17:38.811672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806728    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806732    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806736    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806740    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806745    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806748    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806751    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806754    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806758    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806761    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806764    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806767    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806772    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806776    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806779    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806784    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806787    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806791    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806794    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806797    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806801    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806804    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806807    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:17:38.812176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806810    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806813    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806816    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806819    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806822    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806825    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806829    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806832    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806836    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806838    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806841    2582 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806844    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806848    2582 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806852    2582 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806854    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806857    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806860    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806864    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806867    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806871    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806874    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806878    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806882    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806885    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806888    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:17:38.812735 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806891    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806894    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806897    2582 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806900    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806903    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806906    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806909    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806913    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806916    2582 flags.go:64] FLAG: --help="false"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806918    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-138-105.ec2.internal"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806922    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806925    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806927    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806931    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806934    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806937    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806940    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806942    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806945    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806948    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806951    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806954    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806957    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806959    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:17:38.813368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806962    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806965    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806970    2582 flags.go:64] FLAG: --lock-file=""
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806973    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806977    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806980    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806986    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806989    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806992    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806995    2582 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.806998    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807002    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807004    2582 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807007    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807011    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807014    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807018    2582 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807021    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807024    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807026    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807030    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807032    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807035    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807038    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807045    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:17:38.813958 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807048    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807051    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807054    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807056    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807062    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807066    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807069    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807072    2582 flags.go:64] FLAG: --port="10250"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807075    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807080    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085761312286e075b"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807083    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807086    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807090    2582 flags.go:64] FLAG: --register-node="true"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807093    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807096    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807100    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807103    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807106    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807109    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807112    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807115    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807118    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807121    2582 flags.go:64] FLAG: --runonce="false"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807123    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807127    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:17:38.814614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807129    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807132    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807135    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807139    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807142    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807145    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807147    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807150    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807153    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807156    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807159    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807162    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807165    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807170    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807173    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807175    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807181    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807184    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807187    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807191    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807194    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807197    2582 flags.go:64] FLAG: --v="2"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807202    2582 flags.go:64] FLAG: --version="false"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807206    2582 flags.go:64] FLAG: --vmodule=""
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807210    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:17:38.815329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807213    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807369    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807374    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807377    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807380    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807383    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807386    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807389    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807392    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807395    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807398    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807400    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807403    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807406    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807409    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807412    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807414    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807417    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807419    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807422    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:38.815983 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807424    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807427    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807430    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807434    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807436    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807439    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807443    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807446    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807449    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807451    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807454    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807457    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807460    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807462    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807465    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807468    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807470    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807473    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807475    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807478    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:38.816570 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807480    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807483    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807485    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807488    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807490    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807493    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807496    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807498    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807501    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807504    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807506    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807509    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807511    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807515    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807517    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807521    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807524    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807526    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807530    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807533    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:38.817061 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807536    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807538    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807541    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807543    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807546    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807549    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807551    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807554    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807556    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807560    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807563    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807566    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807569    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807571    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807574    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807576    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807580    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807584    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807587    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:38.817564 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807589    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807592    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807595    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807598    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807601    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807603    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807606    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807608    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807612    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807615    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807617    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807621    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807625    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807629    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807632    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807634    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807637    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807640    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807643    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:38.818028 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807645    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.807648    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.807653    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.815530    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.815544    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.815564    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.815570    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815636    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815640    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815644    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815647    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815650    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815653    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815656    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815658    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:38.818496 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815661    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815664    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815666    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815669    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815671    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815674    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815677    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815680    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815682    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815685    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815687    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815690    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815692    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815695    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815698    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815701    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815703    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815706    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815710    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815713    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:38.818862 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815717    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815721    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815726    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815729    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815732    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815735    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815738    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815741    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815744    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815747    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815750    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815753    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815755    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815759    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815761    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815764    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815767    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815769    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815772    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:38.819363 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815775    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815777    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815780    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815783    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815786    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815788    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815791    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815793    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815796    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815799    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815801    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815804    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815806    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815811    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815815    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815818    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815821    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815824    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815826    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815829    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:38.819821 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815831    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815834    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815837    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815839    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815842    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815844    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815847    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815850    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815853    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815855    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815858    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815861    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815863    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815866    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815869    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815871    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815874    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815877    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815879    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815882    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:38.820379 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815885    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815887    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815890    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815892    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815895    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815898    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815901    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815904    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815907    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815910    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815912    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.815915    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.815920    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816028    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:38.820877 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816034    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816038    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816041    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816044    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816047    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816050    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816053    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816056    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816058    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816061    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816064    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816066    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816069    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816071    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816074    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816076    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816079    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816082    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816084    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816087    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:38.821226 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816090    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816092    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816095    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816097    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816100    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816102    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816105    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816108    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816111    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816114    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816116    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816119    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816121    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816124    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816126    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816129    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816131    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816134    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816136    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816146    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:38.821738 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816148    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816151    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816153    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816156    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816159    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816161    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816164    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816166    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816169    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816172    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816175    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816177    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816180    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816183    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816185    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816188    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816190    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816193    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816195    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816198    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:38.822224 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816201    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816204    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816206    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816209    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816211    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816214    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816216    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816219    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816222    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816224    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816227    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816229    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816232    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816234    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816237    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816240    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816244    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816247    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816250    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:38.822739 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816253    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816271    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816276    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816279    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816281    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816284    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816287    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816289    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816292    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816295    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816297    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816300    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816304    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816307    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816311    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816314    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816317    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816319    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:38.823201 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:38.816322    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:38.823651 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.816326    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:38.823651 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.816427    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:17:38.823651 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.819809    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:17:38.823651 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.820844    2582 server.go:1019] "Starting client certificate rotation"
Sep 05 02:17:38.823651 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.820941    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:38.823651 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.821651    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:38.823825 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.823765    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:38.825816 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.825795    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:38.839029 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.839008    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:17:38.842368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.842352    2582 log.go:25] "Validated CRI v1 image API"
Sep 05 02:17:38.845742 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.845720    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:17:38.853023 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.852996    2582 fs.go:135] Filesystem UUIDs: map[3c3c9180-e9df-459d-8ab8-5ebb70e6f0e9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b773b8e9-474d-4312-b5d2-80eec461c88d:/dev/nvme0n1p3]
Sep 05 02:17:38.853127 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.853018    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:17:38.858229 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.858110    2582 manager.go:216] Machine: {Timestamp:2026-09-05 02:17:38.856714409 +0000 UTC m=+0.377928141 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105312 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2e98f375bfad80b928bde046a6a8 SystemUUID:ec2a2e98-f375-bfad-80b9-28bde046a6a8 BootID:7755f76e-d0c7-421f-9815-a5f18875740d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:57:b8:e3:49:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:57:b8:e3:49:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:87:ba:10:90:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:17:38.858229 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.858219    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:17:38.858384 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.858321    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:17:38.858384 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.858332    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:38.861196 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.861172    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:17:38.861365 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.861199    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:17:38.862106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.862096    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:17:38.862139 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.862110    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:17:38.862139 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.862119    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:38.862964 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.862954    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:38.864242 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.864232    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:17:38.868763 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868751    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:38.868882 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868872    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:17:38.868920 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868903    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:17:38.868920 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868913    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:17:38.868980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868929    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:17:38.868980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868941    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:17:38.868980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.868950    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:17:38.869352 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.869339    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:38.869404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.869364    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:38.873036 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.873020    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:17:38.876354 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.876339    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:17:38.876418 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.876369    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:17:38.877784 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877768    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877791    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877800    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877808    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877824    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877833    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877852    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877860    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877870    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:17:38.877886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877880    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:17:38.878146 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877907    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:17:38.878146 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877916    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:17:38.878146 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877949    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:17:38.878146 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.877957    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:17:38.880058 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.880025    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:17:38.880058 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.880021    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:17:38.880287 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.880249    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:17:38.883179 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.883163    2582 server.go:1268] "Started kubelet"
Sep 05 02:17:38.883272 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.883233    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:17:38.883377 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.883319    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:17:38.883434 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.883402    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:17:38.884036 ip-10-0-138-105 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:17:38.887741 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.887717    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:17:38.889245 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.889217    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:17:38.889509 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.889473    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:17:38.901929 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.901908    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:17:38.902022 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.901971    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:38.902077 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.902045    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:38.902781 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902761    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:17:38.902869 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902788    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:17:38.902869 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902802    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:17:38.902976 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902914    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:17:38.902976 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902930    2582 factory.go:55] Registering systemd factory
Sep 05 02:17:38.902976 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902939    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:17:38.902976 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902949    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:17:38.902976 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.902956    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:17:38.903175 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.903161    2582 factory.go:153] Registering CRI-O factory
Sep 05 02:17:38.903215 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.903180    2582 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:17:38.903215 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.903196    2582 factory.go:103] Registering Raw factory
Sep 05 02:17:38.903215 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.903206    2582 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:17:38.904204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.904188    2582 manager.go:349] Starting recovery of all containers
Sep 05 02:17:38.904310 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.904216    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 02:17:38.904369 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.904340    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 02:17:38.905179 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.903965    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-105.ec2.internal.18d24b2d0cfe97bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-105.ec2.internal,UID:ip-10-0-138-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-105.ec2.internal,},FirstTimestamp:2026-09-05 02:17:38.883131323 +0000 UTC m=+0.404345056,LastTimestamp:2026-09-05 02:17:38.883131323 +0000 UTC m=+0.404345056,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-105.ec2.internal,}"
Sep 05 02:17:38.910218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.910185    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjfsf"
Sep 05 02:17:38.913454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.913439    2582 manager.go:354] Recovery completed
Sep 05 02:17:38.917499 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.917482    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:38.918372 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.918354    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjfsf"
Sep 05 02:17:38.919299 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.919283    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:38.919357 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.919311    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:38.919357 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.919322    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:38.919999 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.919984    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:17:38.919999 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.919998    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:17:38.920162 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.920016    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:38.920162 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.920046    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:38.920858 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.920834    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:38.920920 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.920868    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:38.920920 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.920884    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:38.922047 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.922033    2582 policy_none.go:49] "None policy: Start"
Sep 05 02:17:38.922122 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.922052    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:17:38.922122 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.922066    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:17:38.924370 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.924356    2582 policy_none.go:47] "Start"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.961850    2582 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.961887    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.961896    2582 server.go:87] "Starting device plugin registration server"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.962187    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.962205    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.962330    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.962402    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:38.962410    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.962978    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:17:38.975681 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:38.963029    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:39.002487 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.002454    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:17:39.003628 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.003611    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:17:39.003705 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.003657    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:17:39.003705 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.003681    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:17:39.003775 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.003717    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:17:39.006124 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.006099    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:39.063147 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.063054    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:39.064065 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.064047    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:39.064159 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.064079    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:39.064159 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.064089    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:39.064159 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.064113    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.071533 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.071515    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.071593 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.071541    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-105.ec2.internal\": node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:39.097535 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.097508    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:39.104744 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.104709    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"]
Sep 05 02:17:39.104863 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.104780    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:39.106101 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.106076    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:39.106200 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.106110    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:39.106200 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.106123    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:39.108491 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.108479    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:39.108618 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.108601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.108659 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.108631    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:39.109250 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.109232    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:39.109250 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.109247    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:39.109399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.109278    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:39.109399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.109284    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:39.109399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.109291    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:39.109399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.109298    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:39.111554 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.111539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.111631 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.111562    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:39.112272 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.112243    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:39.112350 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.112279    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:39.112350 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.112290    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:39.141055 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.141030    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-105.ec2.internal\" not found" node="ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.144461 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.144444    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-105.ec2.internal\" not found" node="ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.198591 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.198555    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:39.204634 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.204614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7174215261e78957c7d12a77ffb9679-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal\" (UID: \"b7174215261e78957c7d12a77ffb9679\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.204691 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.204642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7174215261e78957c7d12a77ffb9679-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal\" (UID: \"b7174215261e78957c7d12a77ffb9679\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.204691 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.204659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23947b3ee24ec342ed4ecf585dce8908-config\") pod \"kube-apiserver-proxy-ip-10-0-138-105.ec2.internal\" (UID: \"23947b3ee24ec342ed4ecf585dce8908\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.299408 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.299378    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:39.304748 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.304734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7174215261e78957c7d12a77ffb9679-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal\" (UID: \"b7174215261e78957c7d12a77ffb9679\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.304792 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.304757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23947b3ee24ec342ed4ecf585dce8908-config\") pod \"kube-apiserver-proxy-ip-10-0-138-105.ec2.internal\" (UID: \"23947b3ee24ec342ed4ecf585dce8908\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.304792 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.304776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7174215261e78957c7d12a77ffb9679-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal\" (UID: \"b7174215261e78957c7d12a77ffb9679\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.304885 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.304821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7174215261e78957c7d12a77ffb9679-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal\" (UID: \"b7174215261e78957c7d12a77ffb9679\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.304885 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.304844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7174215261e78957c7d12a77ffb9679-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal\" (UID: \"b7174215261e78957c7d12a77ffb9679\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.304885 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.304874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23947b3ee24ec342ed4ecf585dce8908-config\") pod \"kube-apiserver-proxy-ip-10-0-138-105.ec2.internal\" (UID: \"23947b3ee24ec342ed4ecf585dce8908\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.400293 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:39.400193    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:39.443728 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.443699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:39.447756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:39.447738    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:40.600495 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:40.600461    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:40.700996 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:40.700952    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:40.801520 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:40.801432    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:40.901889 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:40.901860    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:40.920301 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:40.920269    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:17:40.920494 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:40.920468    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:41.002819 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:41.002781    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:41.025125 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.025086    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:12:40 +0000 UTC" deadline="2028-03-30 10:54:38.216634212 +0000 UTC"
Sep 05 02:17:41.025125 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.025118    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13736h36m57.191520142s"
Sep 05 02:17:41.064373 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.064352    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:17:41.103726 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:41.103694    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-105.ec2.internal\" not found"
Sep 05 02:17:41.106996 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.106941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal" event={"ID":"b7174215261e78957c7d12a77ffb9679","Type":"ContainerStarted","Data":"703751f9ebb5108e4bae7fcb7a30f94751f968239d4f9cc2a034fadf6880b2cf"}
Sep 05 02:17:41.108054 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.108034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal" event={"ID":"23947b3ee24ec342ed4ecf585dce8908","Type":"ContainerStarted","Data":"fe78abebbc791f17827eb0857b5856b6f043f259148ab4e347a394af8566b4d9"}
Sep 05 02:17:41.165677 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.165650    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:41.187516 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.187494    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:41.202662 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.202639    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:41.215772 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.215604    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:41.218676 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.218662    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal"
Sep 05 02:17:41.227149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.227131    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:41.286872 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.286845    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:41.970280 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.970250    2582 apiserver.go:52] "Watching apiserver"
Sep 05 02:17:41.980244 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.980217    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:17:41.980635 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.980610    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal","openshift-multus/network-metrics-daemon-qkm56","openshift-ovn-kubernetes/ovnkube-node-njtb5","kube-system/konnectivity-agent-xnglf","openshift-cluster-node-tuning-operator/tuned-clktc","openshift-dns/node-resolver-42lzd","openshift-image-registry/node-ca-zhnn2","openshift-multus/multus-additional-cni-plugins-4hwq4","openshift-multus/multus-k48j2","openshift-network-diagnostics/network-check-target-tvs2z","openshift-network-operator/iptables-alerter-tlncs","kube-system/global-pull-secret-syncer-6rg87","kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"]
Sep 05 02:17:41.983122 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.983101    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:41.985350 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.985327    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:41.985745 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.985713    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:17:41.985745 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.985730    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mfwkn\""
Sep 05 02:17:41.985745 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.985744    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:17:41.986066 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.986043    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:17:41.987617 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.987599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:41.988333 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.988312    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bxmp5\""
Sep 05 02:17:41.988435 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.988333    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:17:41.988658 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.988566    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:41.988852 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.988833    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:41.989891 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.989840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:41.990839 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.990290    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:17:41.990839 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.990360    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vckmw\""
Sep 05 02:17:41.990839 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.990548    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:17:41.990839 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.990642    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:17:41.991062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.990848    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:17:41.991062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.990876    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:17:41.991638 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.991615    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:17:41.992212 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.992197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:41.992623 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.992606    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:17:41.992798 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.992779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zpjmv\""
Sep 05 02:17:41.992925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.992912    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:17:41.994843 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.994756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:41.995205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.995188    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:41.995445 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.995430    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:41.995558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.995542    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqddg\""
Sep 05 02:17:41.996784 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.996765    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5pdfj\""
Sep 05 02:17:41.997089 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.997072    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:17:41.997216 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.997175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k48j2"
Sep 05 02:17:41.997364 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.997347    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:17:41.999980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:41.999948    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:17:42.000073 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.000026    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:17:42.000073 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.000066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l74gn\""
Sep 05 02:17:42.000184 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.000126    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:17:42.000262 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.000245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.000317 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.000270    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:17:42.001942 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.001925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.002114 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.002087    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:42.002927 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.002895    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:42.003218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.003087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2sxtz\""
Sep 05 02:17:42.003410 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.003389    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:17:42.003574 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.003553    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:17:42.003662 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.003618    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:17:42.004559 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.004539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:42.004674 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.004601    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:42.007326 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.007308    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.012140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.011868    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:17:42.012140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.011875    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:17:42.012140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.011953    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4q4hj\""
Sep 05 02:17:42.012897 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-etc-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.012897 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-sys-fs\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.012897 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-systemd-units\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-systemd\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclgn\" (UniqueName: \"kubernetes.io/projected/db1b93ea-3186-4e0e-bde8-720c7ad651e7-kube-api-access-lclgn\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012946    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7f06bf-d675-415e-8a76-1ec169531719-tmp-dir\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.012994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-var-lib-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11061d9f-b38b-4629-a597-6071ab6c5908-ovn-node-metrics-cert\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.013028    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-cni-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.013106 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-daemon-config\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-multus-certs\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-socket-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-run-netns\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-ovnkube-script-lib\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysconfig\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e7f06bf-d675-415e-8a76-1ec169531719-hosts-file\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b8e5e60-cb9f-4685-8abe-ce5d784ded6e-agent-certs\") pod \"konnectivity-agent-xnglf\" (UID: \"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e\") " pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysctl-d\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysctl-conf\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-system-cni-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-k8s-cni-cncf-io\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-kubelet\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.013500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hqp\" (UniqueName: \"kubernetes.io/projected/bd894870-c9ae-4904-a92f-6edba4702fbc-kube-api-access-84hqp\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-ovnkube-config\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446cj\" (UniqueName: \"kubernetes.io/projected/11061d9f-b38b-4629-a597-6071ab6c5908-kube-api-access-446cj\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b8e5e60-cb9f-4685-8abe-ce5d784ded6e-konnectivity-ca\") pod \"konnectivity-agent-xnglf\" (UID: \"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e\") " pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-netns\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-conf-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4h2\" (UniqueName: \"kubernetes.io/projected/a80efa97-46ac-46aa-94f8-187956a1d6a9-kube-api-access-7w4h2\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghsq\" (UniqueName: \"kubernetes.io/projected/3fbe1fd6-5563-4335-a918-05eb171889a9-kube-api-access-cghsq\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-node-log\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-host\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd894870-c9ae-4904-a92f-6edba4702fbc-host\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd894870-c9ae-4904-a92f-6edba4702fbc-serviceca\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-host-slash\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-sys\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.014103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-socket-dir-parent\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-ovn\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.013956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpndk\" (UniqueName: \"kubernetes.io/projected/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-kube-api-access-hpndk\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-slash\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-run-ovn-kubernetes\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-run\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4dh\" (UniqueName: \"kubernetes.io/projected/2e7f06bf-d675-415e-8a76-1ec169531719-kube-api-access-2w4dh\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a80efa97-46ac-46aa-94f8-187956a1d6a9-cni-binary-copy\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-cni-multus\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-hostroot\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-log-socket\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-lib-modules\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-cni-bin\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-iptables-alerter-script\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-cni-bin\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-cnibin\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.014984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-kubelet\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvs8d\" (UniqueName: \"kubernetes.io/projected/9f623e00-8e5e-4eb1-8f53-3751e8474c52-kube-api-access-nvs8d\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-cni-netd\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-modprobe-d\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-systemd\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-var-lib-kubelet\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014722    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-tuned\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-etc-kubernetes\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-device-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-kubernetes\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-registration-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-env-overrides\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.015648 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.014885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db1b93ea-3186-4e0e-bde8-720c7ad651e7-tmp\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.016161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.015926    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:42.041595 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.041576    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gxv8k"
Sep 05 02:17:42.050216 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.050196    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gxv8k"
Sep 05 02:17:42.104321 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.104291    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:17:42.115217 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.115217 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115222    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd894870-c9ae-4904-a92f-6edba4702fbc-host\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd894870-c9ae-4904-a92f-6edba4702fbc-serviceca\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-host-slash\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-sys\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd894870-c9ae-4904-a92f-6edba4702fbc-host\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-socket-dir-parent\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-ovn\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-sys\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpndk\" (UniqueName: \"kubernetes.io/projected/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-kube-api-access-hpndk\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.115390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-slash\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-run-ovn-kubernetes\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-run\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-ovn\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-host-slash\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-run-ovn-kubernetes\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-run\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-slash\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4dh\" (UniqueName: \"kubernetes.io/projected/2e7f06bf-d675-415e-8a76-1ec169531719-kube-api-access-2w4dh\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a80efa97-46ac-46aa-94f8-187956a1d6a9-cni-binary-copy\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-cni-multus\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-hostroot\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-log-socket\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-lib-modules\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-cni-bin\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.115855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-cnibin\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-hostroot\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-cni-binary-copy\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-cni-multus\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-iptables-alerter-script\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-cni-bin\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116029    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd894870-c9ae-4904-a92f-6edba4702fbc-serviceca\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-cni-bin\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-cni-bin\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-cnibin\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-log-socket\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-kubelet\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpmq\" (UniqueName: \"kubernetes.io/projected/c5201b1d-866d-4e87-a351-5e7b24756371-kube-api-access-rzpmq\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvs8d\" (UniqueName: \"kubernetes.io/projected/9f623e00-8e5e-4eb1-8f53-3751e8474c52-kube-api-access-nvs8d\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-cni-netd\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-modprobe-d\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.116571 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-lib-modules\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-systemd\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-var-lib-kubelet\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-var-lib-kubelet\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a80efa97-46ac-46aa-94f8-187956a1d6a9-cni-binary-copy\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-tuned\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-var-lib-kubelet\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-cni-netd\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-cnibin\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-systemd\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-modprobe-d\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-iptables-alerter-script\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-etc-kubernetes\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-etc-kubernetes\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-device-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-kubernetes\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-registration-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-env-overrides\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.117404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db1b93ea-3186-4e0e-bde8-720c7ad651e7-tmp\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-registration-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-kubernetes\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-etc-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-etc-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-sys-fs\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-systemd-units\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.115587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-socket-dir-parent\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.116983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-systemd\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclgn\" (UniqueName: \"kubernetes.io/projected/db1b93ea-3186-4e0e-bde8-720c7ad651e7-kube-api-access-lclgn\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-device-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7f06bf-d675-415e-8a76-1ec169531719-tmp-dir\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-system-cni-dir\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-systemd\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-var-lib-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11061d9f-b38b-4629-a597-6071ab6c5908-ovn-node-metrics-cert\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.118263 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-sys-fs\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-cni-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-daemon-config\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-multus-certs\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117261    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-socket-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-cni-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-run-netns\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-ovnkube-script-lib\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysconfig\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e7f06bf-d675-415e-8a76-1ec169531719-hosts-file\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7f06bf-d675-415e-8a76-1ec169531719-tmp-dir\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-dbus\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-var-lib-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-env-overrides\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b8e5e60-cb9f-4685-8abe-ce5d784ded6e-agent-certs\") pod \"konnectivity-agent-xnglf\" (UID: \"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e\") " pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.119061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysctl-d\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-multus-certs\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-systemd-units\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysctl-conf\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysctl-d\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-system-cni-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-k8s-cni-cncf-io\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysctl-conf\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-run-netns\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-daemon-config\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-kubelet\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-system-cni-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-sysconfig\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.119895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.117893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-socket-dir\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-kubelet-config\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e7f06bf-d675-415e-8a76-1ec169531719-hosts-file\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3fbe1fd6-5563-4335-a918-05eb171889a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-ovnkube-script-lib\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-host-kubelet\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-k8s-cni-cncf-io\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hqp\" (UniqueName: \"kubernetes.io/projected/bd894870-c9ae-4904-a92f-6edba4702fbc-kube-api-access-84hqp\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.118207    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.118313    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.618284048 +0000 UTC m=+3.039789383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-ovnkube-config\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446cj\" (UniqueName: \"kubernetes.io/projected/11061d9f-b38b-4629-a597-6071ab6c5908-kube-api-access-446cj\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b8e5e60-cb9f-4685-8abe-ce5d784ded6e-konnectivity-ca\") pod \"konnectivity-agent-xnglf\" (UID: \"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e\") " pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-netns\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-host-run-netns\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-conf-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.120668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4h2\" (UniqueName: \"kubernetes.io/projected/a80efa97-46ac-46aa-94f8-187956a1d6a9-kube-api-access-7w4h2\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghsq\" (UniqueName: \"kubernetes.io/projected/3fbe1fd6-5563-4335-a918-05eb171889a9-kube-api-access-cghsq\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-node-log\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-host\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-run-openvswitch\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118746    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/11061d9f-b38b-4629-a597-6071ab6c5908-node-log\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db1b93ea-3186-4e0e-bde8-720c7ad651e7-host\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a80efa97-46ac-46aa-94f8-187956a1d6a9-multus-conf-dir\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.118903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/11061d9f-b38b-4629-a597-6071ab6c5908-ovnkube-config\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.121449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.119314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2b8e5e60-cb9f-4685-8abe-ce5d784ded6e-konnectivity-ca\") pod \"konnectivity-agent-xnglf\" (UID: \"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e\") " pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.121794 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.121667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db1b93ea-3186-4e0e-bde8-720c7ad651e7-tmp\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.121794 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.121736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db1b93ea-3186-4e0e-bde8-720c7ad651e7-etc-tuned\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.121888 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.121871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/11061d9f-b38b-4629-a597-6071ab6c5908-ovn-node-metrics-cert\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.122109 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.122053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2b8e5e60-cb9f-4685-8abe-ce5d784ded6e-agent-certs\") pod \"konnectivity-agent-xnglf\" (UID: \"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e\") " pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.129304 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.128753    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:42.129304 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.128780    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:42.129304 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.128794    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.129304 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.129023    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.628985539 +0000 UTC m=+3.050490860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.131989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.130328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvs8d\" (UniqueName: \"kubernetes.io/projected/9f623e00-8e5e-4eb1-8f53-3751e8474c52-kube-api-access-nvs8d\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.131989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.130577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghsq\" (UniqueName: \"kubernetes.io/projected/3fbe1fd6-5563-4335-a918-05eb171889a9-kube-api-access-cghsq\") pod \"aws-ebs-csi-driver-node-f8nrf\" (UID: \"3fbe1fd6-5563-4335-a918-05eb171889a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.133989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.133941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclgn\" (UniqueName: \"kubernetes.io/projected/db1b93ea-3186-4e0e-bde8-720c7ad651e7-kube-api-access-lclgn\") pod \"tuned-clktc\" (UID: \"db1b93ea-3186-4e0e-bde8-720c7ad651e7\") " pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.134174 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.134019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4h2\" (UniqueName: \"kubernetes.io/projected/a80efa97-46ac-46aa-94f8-187956a1d6a9-kube-api-access-7w4h2\") pod \"multus-k48j2\" (UID: \"a80efa97-46ac-46aa-94f8-187956a1d6a9\") " pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.134174 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.134029    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4dh\" (UniqueName: \"kubernetes.io/projected/2e7f06bf-d675-415e-8a76-1ec169531719-kube-api-access-2w4dh\") pod \"node-resolver-42lzd\" (UID: \"2e7f06bf-d675-415e-8a76-1ec169531719\") " pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.134528 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.134506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-446cj\" (UniqueName: \"kubernetes.io/projected/11061d9f-b38b-4629-a597-6071ab6c5908-kube-api-access-446cj\") pod \"ovnkube-node-njtb5\" (UID: \"11061d9f-b38b-4629-a597-6071ab6c5908\") " pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.135514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.135488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hqp\" (UniqueName: \"kubernetes.io/projected/bd894870-c9ae-4904-a92f-6edba4702fbc-kube-api-access-84hqp\") pod \"node-ca-zhnn2\" (UID: \"bd894870-c9ae-4904-a92f-6edba4702fbc\") " pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.136647 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.136593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpndk\" (UniqueName: \"kubernetes.io/projected/93920abd-4a9c-45ac-805b-98dc5e2ff6ff-kube-api-access-hpndk\") pod \"iptables-alerter-tlncs\" (UID: \"93920abd-4a9c-45ac-805b-98dc5e2ff6ff\") " pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.219327 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219327 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-kubelet-config\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.219541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-cnibin\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-cni-binary-copy\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpmq\" (UniqueName: \"kubernetes.io/projected/c5201b1d-866d-4e87-a351-5e7b24756371-kube-api-access-rzpmq\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-cnibin\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-kubelet-config\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-system-cni-dir\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-dbus\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-system-cni-dir\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.219760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5201b1d-866d-4e87-a351-5e7b24756371-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.220018 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.219747    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.220018 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-dbus\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.220018 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.219855    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.719835894 +0000 UTC m=+3.141341220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.220018 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.219901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.220146 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.220103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-cni-binary-copy\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.220178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.220155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5201b1d-866d-4e87-a351-5e7b24756371-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.229342 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.229284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpmq\" (UniqueName: \"kubernetes.io/projected/c5201b1d-866d-4e87-a351-5e7b24756371-kube-api-access-rzpmq\") pod \"multus-additional-cni-plugins-4hwq4\" (UID: \"c5201b1d-866d-4e87-a351-5e7b24756371\") " pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.269565 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.269539    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:42.294679 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.294643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zhnn2"
Sep 05 02:17:42.302224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.302206    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlncs"
Sep 05 02:17:42.315220 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.315197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:17:42.318278 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.318256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:17:42.324997 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.324978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-clktc"
Sep 05 02:17:42.330732 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.330715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42lzd"
Sep 05 02:17:42.337550 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.337530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k48j2"
Sep 05 02:17:42.344126 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.344107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf"
Sep 05 02:17:42.349695 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.349678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4hwq4"
Sep 05 02:17:42.622672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.622581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:42.622844 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.622721    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.622844 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.622787    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:43.622770276 +0000 UTC m=+4.044275596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.723432 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.723399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:42.723620 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:42.723462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:42.723620 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.723588    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.723717 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.723656    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:43.723638642 +0000 UTC m=+4.145143982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.723717 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.723589    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:42.723717 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.723701    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:42.723717 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.723713    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.723876 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:42.723761    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:43.723746051 +0000 UTC m=+4.145251399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.917901 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.917848    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8e5e60_cb9f_4685_8abe_ce5d784ded6e.slice/crio-2faba2093ca1f0665e3414c5ef418046f987a481a78f3c667dd4b164109ee5e8 WatchSource:0}: Error finding container 2faba2093ca1f0665e3414c5ef418046f987a481a78f3c667dd4b164109ee5e8: Status 404 returned error can't find the container with id 2faba2093ca1f0665e3414c5ef418046f987a481a78f3c667dd4b164109ee5e8
Sep 05 02:17:42.920672 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.920627    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5201b1d_866d_4e87_a351_5e7b24756371.slice/crio-57ede609bfd87aa5f71fdfb099626dd1a4919344def13ea3460e4d7d3891933f WatchSource:0}: Error finding container 57ede609bfd87aa5f71fdfb099626dd1a4919344def13ea3460e4d7d3891933f: Status 404 returned error can't find the container with id 57ede609bfd87aa5f71fdfb099626dd1a4919344def13ea3460e4d7d3891933f
Sep 05 02:17:42.921293 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.921230    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7f06bf_d675_415e_8a76_1ec169531719.slice/crio-ee8d95cdd11025ae147ba03c5bdc34405431ade79e9fcf05f769e3f2343d271a WatchSource:0}: Error finding container ee8d95cdd11025ae147ba03c5bdc34405431ade79e9fcf05f769e3f2343d271a: Status 404 returned error can't find the container with id ee8d95cdd11025ae147ba03c5bdc34405431ade79e9fcf05f769e3f2343d271a
Sep 05 02:17:42.924163 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.924142    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80efa97_46ac_46aa_94f8_187956a1d6a9.slice/crio-303b8b45cbf9f62cb52dad59fcbfeb5986cc3cc1248bf4bb638cdfc2abe9b78d WatchSource:0}: Error finding container 303b8b45cbf9f62cb52dad59fcbfeb5986cc3cc1248bf4bb638cdfc2abe9b78d: Status 404 returned error can't find the container with id 303b8b45cbf9f62cb52dad59fcbfeb5986cc3cc1248bf4bb638cdfc2abe9b78d
Sep 05 02:17:42.925790 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.925759    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93920abd_4a9c_45ac_805b_98dc5e2ff6ff.slice/crio-ec27355ee148328da4e78d62b63a926ceede798f2a19a7743cfcd49062e65a28 WatchSource:0}: Error finding container ec27355ee148328da4e78d62b63a926ceede798f2a19a7743cfcd49062e65a28: Status 404 returned error can't find the container with id ec27355ee148328da4e78d62b63a926ceede798f2a19a7743cfcd49062e65a28
Sep 05 02:17:42.926713 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.926686    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd894870_c9ae_4904_a92f_6edba4702fbc.slice/crio-4a58b7cb37f7a0667594f801f2a4881677e814046dc59ab93c7672405f383c52 WatchSource:0}: Error finding container 4a58b7cb37f7a0667594f801f2a4881677e814046dc59ab93c7672405f383c52: Status 404 returned error can't find the container with id 4a58b7cb37f7a0667594f801f2a4881677e814046dc59ab93c7672405f383c52
Sep 05 02:17:42.927635 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.927381    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbe1fd6_5563_4335_a918_05eb171889a9.slice/crio-2ef99bf2001fa5c9153e796924f7f1bee7090a6489d90050380423cd28a2dc2c WatchSource:0}: Error finding container 2ef99bf2001fa5c9153e796924f7f1bee7090a6489d90050380423cd28a2dc2c: Status 404 returned error can't find the container with id 2ef99bf2001fa5c9153e796924f7f1bee7090a6489d90050380423cd28a2dc2c
Sep 05 02:17:42.928562 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.928540    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11061d9f_b38b_4629_a597_6071ab6c5908.slice/crio-41406a83d99e3822396e94a9732258d0359dabb0770fe2b2f607a1c976f222bd WatchSource:0}: Error finding container 41406a83d99e3822396e94a9732258d0359dabb0770fe2b2f607a1c976f222bd: Status 404 returned error can't find the container with id 41406a83d99e3822396e94a9732258d0359dabb0770fe2b2f607a1c976f222bd
Sep 05 02:17:42.930548 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:17:42.930524    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1b93ea_3186_4e0e_bde8_720c7ad651e7.slice/crio-2e885d909ef1d84e68d6c3e48318060d28b1b019afbe81c9eb6a9ff55ffb4d34 WatchSource:0}: Error finding container 2e885d909ef1d84e68d6c3e48318060d28b1b019afbe81c9eb6a9ff55ffb4d34: Status 404 returned error can't find the container with id 2e885d909ef1d84e68d6c3e48318060d28b1b019afbe81c9eb6a9ff55ffb4d34
Sep 05 02:17:43.053662 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.053601    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:42 +0000 UTC" deadline="2028-05-16 07:34:34.094552154 +0000 UTC"
Sep 05 02:17:43.053662 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.053659    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14861h16m51.040897626s"
Sep 05 02:17:43.104064 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.104016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:43.104219 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.104156    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:43.111805 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.111769    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"41406a83d99e3822396e94a9732258d0359dabb0770fe2b2f607a1c976f222bd"}
Sep 05 02:17:43.113064 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.113036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlncs" event={"ID":"93920abd-4a9c-45ac-805b-98dc5e2ff6ff","Type":"ContainerStarted","Data":"ec27355ee148328da4e78d62b63a926ceede798f2a19a7743cfcd49062e65a28"}
Sep 05 02:17:43.114257 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.114236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerStarted","Data":"57ede609bfd87aa5f71fdfb099626dd1a4919344def13ea3460e4d7d3891933f"}
Sep 05 02:17:43.115476 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.115435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnglf" event={"ID":"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e","Type":"ContainerStarted","Data":"2faba2093ca1f0665e3414c5ef418046f987a481a78f3c667dd4b164109ee5e8"}
Sep 05 02:17:43.117497 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.117473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal" event={"ID":"23947b3ee24ec342ed4ecf585dce8908","Type":"ContainerStarted","Data":"ae3abb7c7aca5d03ad395426f6548b1f7d4423ffabb3625a53411dbbaa5ffaf2"}
Sep 05 02:17:43.118762 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.118734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf" event={"ID":"3fbe1fd6-5563-4335-a918-05eb171889a9","Type":"ContainerStarted","Data":"2ef99bf2001fa5c9153e796924f7f1bee7090a6489d90050380423cd28a2dc2c"}
Sep 05 02:17:43.119954 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.119911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zhnn2" event={"ID":"bd894870-c9ae-4904-a92f-6edba4702fbc","Type":"ContainerStarted","Data":"4a58b7cb37f7a0667594f801f2a4881677e814046dc59ab93c7672405f383c52"}
Sep 05 02:17:43.121227 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.121209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k48j2" event={"ID":"a80efa97-46ac-46aa-94f8-187956a1d6a9","Type":"ContainerStarted","Data":"303b8b45cbf9f62cb52dad59fcbfeb5986cc3cc1248bf4bb638cdfc2abe9b78d"}
Sep 05 02:17:43.123371 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.123352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42lzd" event={"ID":"2e7f06bf-d675-415e-8a76-1ec169531719","Type":"ContainerStarted","Data":"ee8d95cdd11025ae147ba03c5bdc34405431ade79e9fcf05f769e3f2343d271a"}
Sep 05 02:17:43.125921 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.125899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clktc" event={"ID":"db1b93ea-3186-4e0e-bde8-720c7ad651e7","Type":"ContainerStarted","Data":"2e885d909ef1d84e68d6c3e48318060d28b1b019afbe81c9eb6a9ff55ffb4d34"}
Sep 05 02:17:43.131328 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.131286    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-105.ec2.internal" podStartSLOduration=2.131273158 podStartE2EDuration="2.131273158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:43.131225441 +0000 UTC m=+3.552730771" watchObservedRunningTime="2026-09-05 02:17:43.131273158 +0000 UTC m=+3.552778497"
Sep 05 02:17:43.631593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.631503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:43.631765 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.631669    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:43.631765 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.631733    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:45.631714075 +0000 UTC m=+6.053219399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.732759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:43.732830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.732998    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.733060    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:45.733042835 +0000 UTC m=+6.154548159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.733493    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.733514    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.733527    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:43.733606 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:43.733570    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:45.73355602 +0000 UTC m=+6.155061351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:44.054833 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:44.054713    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:42 +0000 UTC" deadline="2028-06-17 21:13:14.084188434 +0000 UTC"
Sep 05 02:17:44.054833 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:44.054779    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15642h55m30.029413121s"
Sep 05 02:17:44.107403 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:44.106353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:44.107403 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:44.106476    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:44.107403 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:44.106849    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:44.107403 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:44.106958    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:44.143369 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:44.143331    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7174215261e78957c7d12a77ffb9679" containerID="961abbe152eeaa2bbaea3008fae1799d661ad5473fa7d8ad08b9a345245ae267" exitCode=0
Sep 05 02:17:44.143921 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:44.143893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal" event={"ID":"b7174215261e78957c7d12a77ffb9679","Type":"ContainerDied","Data":"961abbe152eeaa2bbaea3008fae1799d661ad5473fa7d8ad08b9a345245ae267"}
Sep 05 02:17:45.104186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:45.103825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:45.104634 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.104373    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:45.179088 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:45.179048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal" event={"ID":"b7174215261e78957c7d12a77ffb9679","Type":"ContainerStarted","Data":"d7ccc822dd8a173fcdb158a096dec03f806543b236b78a676019af1e26bd8292"}
Sep 05 02:17:45.655980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:45.655916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:45.656176 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.656129    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:45.656241 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.656193    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:49.656174562 +0000 UTC m=+10.077679883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:45.756319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:45.756386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.756531    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.756595    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:49.756577078 +0000 UTC m=+10.178082400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.757010    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.757032    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.757044    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:45.757128 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:45.757089    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:49.7570737 +0000 UTC m=+10.178579031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:46.104139 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:46.104058    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:46.104300 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:46.104232    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:46.104300 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:46.104291    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:46.104704 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:46.104376    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:47.103616 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:47.103580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:47.103801 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:47.103717    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:48.104300 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:48.104262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:48.104804 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:48.104412    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:48.104804 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:48.104262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:48.104944 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:48.104873    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:49.104665 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:49.104423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:49.104665 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.104562    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:49.692747 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:49.692709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:49.692932 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.692888    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:49.693036 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.692958    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:57.692937955 +0000 UTC m=+18.114443278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:49.794076 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:49.794034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:49.794253 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:49.794173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:49.794253 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.794190    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:49.794253 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.794254    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:57.794234613 +0000 UTC m=+18.215739932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:49.794428 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.794281    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:49.794428 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.794297    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:49.794428 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.794306    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:49.794428 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:49.794350    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:57.794337646 +0000 UTC m=+18.215842964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:50.105730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:50.105102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:50.105730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:50.105201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:50.105730 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:50.105294    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:50.105730 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:50.105337    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:51.104236 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:51.104198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:51.104409 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:51.104336    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:52.103995 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:52.103947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:52.104385 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:52.104096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:52.104385 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:52.104164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:52.104385 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:52.104268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:53.104358 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:53.104315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:53.104713 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:53.104428    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:54.104178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:54.104143    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:54.104178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:54.104163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:54.104709 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:54.104263    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:54.104709 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:54.104414    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:55.103744 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:55.103706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:55.103904 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:55.103840    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:56.104313 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:56.104272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:56.104791 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:56.104321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:56.104791 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:56.104417    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:56.104791 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:56.104577    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:57.104073 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:57.104037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:57.104253 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.104148    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:17:57.753734 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:57.753699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:57.754185 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.753858    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:57.754185 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.753937    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.753914056 +0000 UTC m=+34.175419376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:57.854668 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:57.854630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:57.854888 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:57.854742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:57.854888 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.854815    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:57.854888 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.854866    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:57.854888 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.854882    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:57.855077 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.854907    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:57.855077 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.854892    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.85487191 +0000 UTC m=+34.276377226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:57.855077 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:57.854986    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.85495558 +0000 UTC m=+34.276460901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:58.104633 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:58.104523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:17:58.104786 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:58.104666    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:17:58.104786 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:58.104535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:17:58.104786 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:58.104765    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:17:59.104476 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:17:59.104441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:17:59.104934 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:17:59.104565    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:00.105217 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:00.105179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:00.105620 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:00.105296    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:00.105620 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:00.105376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:00.105620 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:00.105510    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:01.104378 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.104354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:01.104470 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:01.104453    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:01.214554 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.214353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf" event={"ID":"3fbe1fd6-5563-4335-a918-05eb171889a9","Type":"ContainerStarted","Data":"2637ff7aabac150dafedf2eeaae38e0ccee3f01da5ebfc3194b156a56334837d"}
Sep 05 02:18:01.215921 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.215888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zhnn2" event={"ID":"bd894870-c9ae-4904-a92f-6edba4702fbc","Type":"ContainerStarted","Data":"c7e46465a7761c900027f531deebe7db1cd4b5f26948b8a0fbe1e23fbf6f3b59"}
Sep 05 02:18:01.217331 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.217308    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k48j2" event={"ID":"a80efa97-46ac-46aa-94f8-187956a1d6a9","Type":"ContainerStarted","Data":"ea66aa48e5b866603b12fb7d8ab8e397a3b741f59a13dd7abb2e3de2d1536fda"}
Sep 05 02:18:01.218845 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.218823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42lzd" event={"ID":"2e7f06bf-d675-415e-8a76-1ec169531719","Type":"ContainerStarted","Data":"0b9838b70fc5e0fa10f4c3fee66b0e880f6c973d0faedf4b09734321348f67f3"}
Sep 05 02:18:01.220432 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.220411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-clktc" event={"ID":"db1b93ea-3186-4e0e-bde8-720c7ad651e7","Type":"ContainerStarted","Data":"0a66f8b66c0734d4dafb95d200f74dc04447640e2271eeb258918d84672dc5db"}
Sep 05 02:18:01.225845 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.225821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"5a4351aee8969b8d180eceb0d7965d9c29b5ef67fab1c776f1f51b85837ff9f1"}
Sep 05 02:18:01.225991 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.225925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"decfa7c12754a0d85c288f460b7ebf398c76003a9909e59a6880da80d7e3c48a"}
Sep 05 02:18:01.226062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.225997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"5aac2c623ede53ee6f2781a19d6798c28a39d96211dc323bdca7617c8ea40c11"}
Sep 05 02:18:01.226062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.226047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"d0c8d63ceb396a8ca75764a4e984dbbdae1dca4ec97764d925e8cd77eca329a7"}
Sep 05 02:18:01.226160 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.226103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"b32ff2d063b80adafd9621e2daa85e5bb5e69be56122ff3c749fbf8550937379"}
Sep 05 02:18:01.227333 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.227311    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5201b1d-866d-4e87-a351-5e7b24756371" containerID="0bc5834ba25ccf5e250c418cb17d1b0b635469dd055a9d27d7b9a4c90fb5182d" exitCode=0
Sep 05 02:18:01.227422 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.227390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerDied","Data":"0bc5834ba25ccf5e250c418cb17d1b0b635469dd055a9d27d7b9a4c90fb5182d"}
Sep 05 02:18:01.228858 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.228835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnglf" event={"ID":"2b8e5e60-cb9f-4685-8abe-ce5d784ded6e","Type":"ContainerStarted","Data":"d170546a539343dcd8d7fbd001f28df21a9c87f9ac9383a9ec6742680e904d2d"}
Sep 05 02:18:01.230842 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.230811    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-105.ec2.internal" podStartSLOduration=20.230800879 podStartE2EDuration="20.230800879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:45.195432967 +0000 UTC m=+5.616938307" watchObservedRunningTime="2026-09-05 02:18:01.230800879 +0000 UTC m=+21.652306218"
Sep 05 02:18:01.231166 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.231144    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zhnn2" podStartSLOduration=3.424614137 podStartE2EDuration="21.231138131s" totalImagesPullingTime="17.806523994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:01.230771732 +0000 UTC m=+21.652277072" watchObservedRunningTime="2026-09-05 02:18:01.231138131 +0000 UTC m=+21.652643469"
Sep 05 02:18:01.248061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.248024    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k48j2" podStartSLOduration=3.350242887 podStartE2EDuration="21.248010519s" totalImagesPullingTime="17.897767632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:01.247956474 +0000 UTC m=+21.669461813" watchObservedRunningTime="2026-09-05 02:18:01.248010519 +0000 UTC m=+21.669515860"
Sep 05 02:18:01.285514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.285463    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xnglf" podStartSLOduration=7.259215191 podStartE2EDuration="21.28544795s" totalImagesPullingTime="14.026232759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:01.285329397 +0000 UTC m=+21.706834735" watchObservedRunningTime="2026-09-05 02:18:01.28544795 +0000 UTC m=+21.706953267"
Sep 05 02:18:01.303363 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.303322    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-clktc" podStartSLOduration=3.4988019 podStartE2EDuration="21.303309192s" totalImagesPullingTime="17.804507292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:01.302808793 +0000 UTC m=+21.724314133" watchObservedRunningTime="2026-09-05 02:18:01.303309192 +0000 UTC m=+21.724814531"
Sep 05 02:18:01.318236 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.318195    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-42lzd" podStartSLOduration=3.521239312 podStartE2EDuration="21.318181141s" totalImagesPullingTime="17.796941829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:01.318120014 +0000 UTC m=+21.739625355" watchObservedRunningTime="2026-09-05 02:18:01.318181141 +0000 UTC m=+21.739686479"
Sep 05 02:18:01.329438 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.329412    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:18:01.330044 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:01.330025    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:18:02.104404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.104290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:02.104604 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:02.104429    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:02.104604 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.104447    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:02.104604 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:02.104542    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:02.234878 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.234835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"217094c0a93197fd9aac1e90f0509d36cc1fa109ffa0ac7100667dea56fdc9a1"}
Sep 05 02:18:02.236625 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.236593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlncs" event={"ID":"93920abd-4a9c-45ac-805b-98dc5e2ff6ff","Type":"ContainerStarted","Data":"20dc1016a7fc61634b46af301607a953acd38096ab7ab2008f1d877769c539d1"}
Sep 05 02:18:02.237259 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.237234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:18:02.237506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.237492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnglf"
Sep 05 02:18:02.257723 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.257676    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tlncs" podStartSLOduration=4.44973253 podStartE2EDuration="22.257660225s" totalImagesPullingTime="17.807927695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:02.257323508 +0000 UTC m=+22.678828846" watchObservedRunningTime="2026-09-05 02:18:02.257660225 +0000 UTC m=+22.679165564"
Sep 05 02:18:02.448724 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:02.448695    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:03.078267 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:03.078150    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:02.448723764Z","UUID":"843a1bb4-5668-4773-96f2-1b2f8d5442bc","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:03.079918 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:03.079893    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:03.080082 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:03.079932    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:03.104386 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:03.104350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:03.104529 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:03.104449    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:03.241223 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:03.241186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf" event={"ID":"3fbe1fd6-5563-4335-a918-05eb171889a9","Type":"ContainerStarted","Data":"6b99279d9aff847f68275c02b307a904233d7c90353fd16c160d1edbbcc7a298"}
Sep 05 02:18:04.104602 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:04.104525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:04.104766 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:04.104651    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:04.104766 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:04.104682    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:04.104878 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:04.104794    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:04.247235 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:04.247204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"9186b53ae8c9a7150fd4baa04a20aef266b211b5532882053a1224705dcc578b"}
Sep 05 02:18:04.249511 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:04.249470    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf" event={"ID":"3fbe1fd6-5563-4335-a918-05eb171889a9","Type":"ContainerStarted","Data":"09f4c140e4d653569e5363ef76c7a1161f67f5c206e9841ccd9ca8d26ff05885"}
Sep 05 02:18:04.270598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:04.270540    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f8nrf" podStartSLOduration=4.082245234 podStartE2EDuration="24.270521648s" totalImagesPullingTime="20.188276414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:04.269506842 +0000 UTC m=+24.691012182" watchObservedRunningTime="2026-09-05 02:18:04.270521648 +0000 UTC m=+24.692026991"
Sep 05 02:18:05.104022 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:05.103983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:05.104225 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:05.104124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:06.107491 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.107456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:06.108149 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:06.107551    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:06.108149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.107603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:06.108149 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:06.107709    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:06.256481 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.256448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" event={"ID":"11061d9f-b38b-4629-a597-6071ab6c5908","Type":"ContainerStarted","Data":"e64e90d5c71db6526ba5a0cd6a62e690994aaba2207988209b680dca8cb0f20b"}
Sep 05 02:18:06.256799 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.256677    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:18:06.256799 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.256751    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:18:06.258324 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.258300    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5201b1d-866d-4e87-a351-5e7b24756371" containerID="d8005b9946a867f97062e60bfe02e75b27b5bc643f28cdc9ee142e84f35be41c" exitCode=0
Sep 05 02:18:06.258426 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.258347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerDied","Data":"d8005b9946a867f97062e60bfe02e75b27b5bc643f28cdc9ee142e84f35be41c"}
Sep 05 02:18:06.271956 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.271928    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:18:06.282676 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:06.282633    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5" podStartSLOduration=8.440890199 podStartE2EDuration="26.28261581s" totalImagesPullingTime="17.841725611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:06.28259732 +0000 UTC m=+26.704102662" watchObservedRunningTime="2026-09-05 02:18:06.28261581 +0000 UTC m=+26.704121146"
Sep 05 02:18:07.104091 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.104059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:07.104217 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:07.104157    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:07.261264 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.261010    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:18:07.276794 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.276768    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:18:07.507654 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.507532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6rg87"]
Sep 05 02:18:07.507834 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.507689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:07.507834 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:07.507782    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:07.511883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.511726    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvs2z"]
Sep 05 02:18:07.511883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.511833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:07.512074 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:07.511928    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:07.512453 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.512434    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qkm56"]
Sep 05 02:18:07.512535 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:07.512525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:07.512612 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:07.512598    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:08.265507 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:08.265476    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5201b1d-866d-4e87-a351-5e7b24756371" containerID="299e168e4164495005584f8ea478943363ec4cbb8475b05a1c63b652c1474822" exitCode=0
Sep 05 02:18:08.266005 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:08.265565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerDied","Data":"299e168e4164495005584f8ea478943363ec4cbb8475b05a1c63b652c1474822"}
Sep 05 02:18:09.104170 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:09.104135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:09.104397 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:09.104238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:09.104397 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:09.104247    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:09.104397 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:09.104273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:09.104397 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:09.104375    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:09.104592 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:09.104445    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:10.272878 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:10.272839    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5201b1d-866d-4e87-a351-5e7b24756371" containerID="48f812e1079413d174508f597cb03ea1ef3032c782dbc986d719140b33a2295b" exitCode=0
Sep 05 02:18:10.273322 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:10.272896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerDied","Data":"48f812e1079413d174508f597cb03ea1ef3032c782dbc986d719140b33a2295b"}
Sep 05 02:18:11.104006 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:11.103954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:11.104201 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:11.103954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:11.104201 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:11.104104    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:11.104341 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:11.104202    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:11.104341 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:11.103954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:11.104341 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:11.104314    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:13.103613 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.103576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:13.103613 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.103615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:13.104243 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.103671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:13.104243 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.103687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tvs2z" podUID="01e674ec-92bd-4c4d-9efc-78b1268bd8e8"
Sep 05 02:18:13.104243 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.103778    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qkm56" podUID="9f623e00-8e5e-4eb1-8f53-3751e8474c52"
Sep 05 02:18:13.104243 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.103867    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rg87" podUID="676f3777-f8f1-4dcc-bf7b-6f95e34c07e2"
Sep 05 02:18:13.377992 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.377902    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-105.ec2.internal" event="NodeReady"
Sep 05 02:18:13.378153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.378077    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:13.415334 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.415232    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c554659c7-k9ljc"]
Sep 05 02:18:13.417584 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.417554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.420064 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.420040    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:18:13.420186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.420170    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f79x9\""
Sep 05 02:18:13.420186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.420179    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:18:13.420610 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.420589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:18:13.425379 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.425358    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:18:13.429011 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.428989    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mgvxp"]
Sep 05 02:18:13.430636 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.430621    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.432739 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.432690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c554659c7-k9ljc"]
Sep 05 02:18:13.433025 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.433001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl7gt\""
Sep 05 02:18:13.433217 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.433155    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:13.433217 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.433174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:13.435483 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.435451    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-98vk8"]
Sep 05 02:18:13.438521 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.437894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.440699 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.440680    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:13.440806 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.440785    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cd6qr\""
Sep 05 02:18:13.440866 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.440803    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:13.440917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.440868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:13.445000 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.444882    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgvxp"]
Sep 05 02:18:13.447560 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.447512    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-98vk8"]
Sep 05 02:18:13.567596 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-tmp-dir\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.567749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-trusted-ca\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.567749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5740d722-bab5-4ed8-9085-f5689745880b-cert\") pod \"ingress-canary-98vk8\" (UID: \"5740d722-bab5-4ed8-9085-f5689745880b\") " pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.567749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.567749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-ca-trust-extracted\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.567749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-installation-pull-secrets\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.567989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-bound-sa-token\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.567989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-config-volume\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.567989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.567989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9vrp\" (UniqueName: \"kubernetes.io/projected/5740d722-bab5-4ed8-9085-f5689745880b-kube-api-access-w9vrp\") pod \"ingress-canary-98vk8\" (UID: \"5740d722-bab5-4ed8-9085-f5689745880b\") " pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.567989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zjm\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-kube-api-access-49zjm\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.568198 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.567990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-certificates\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.568198 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.568033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5mf\" (UniqueName: \"kubernetes.io/projected/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-kube-api-access-gp5mf\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.568198 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.568073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-image-registry-private-configuration\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.668950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.668858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zjm\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-kube-api-access-49zjm\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.668950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.668921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-certificates\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.668952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mf\" (UniqueName: \"kubernetes.io/projected/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-kube-api-access-gp5mf\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.668996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-image-registry-private-configuration\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-tmp-dir\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-trusted-ca\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5740d722-bab5-4ed8-9085-f5689745880b-cert\") pod \"ingress-canary-98vk8\" (UID: \"5740d722-bab5-4ed8-9085-f5689745880b\") " pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-ca-trust-extracted\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-installation-pull-secrets\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669486 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-bound-sa-token\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669486 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-config-volume\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.669486 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669486 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9vrp\" (UniqueName: \"kubernetes.io/projected/5740d722-bab5-4ed8-9085-f5689745880b-kube-api-access-w9vrp\") pod \"ingress-canary-98vk8\" (UID: \"5740d722-bab5-4ed8-9085-f5689745880b\") " pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.669486 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-tmp-dir\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.669715 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.669648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-certificates\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.669715 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.669667    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:13.669819 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.669786    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:14.16976254 +0000 UTC m=+34.591267857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:13.669889 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.669867    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:13.669889 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.669885    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:13.670031 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.669931    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:18:14.169915624 +0000 UTC m=+34.591420958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:13.670168 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.670145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-config-volume\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.670226 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.670184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-trusted-ca\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.670388 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.670348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-ca-trust-extracted\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.674504 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.674361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-image-registry-private-configuration\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.674622 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.674360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-installation-pull-secrets\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.674622 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.674368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5740d722-bab5-4ed8-9085-f5689745880b-cert\") pod \"ingress-canary-98vk8\" (UID: \"5740d722-bab5-4ed8-9085-f5689745880b\") " pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.680712 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.680667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5mf\" (UniqueName: \"kubernetes.io/projected/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-kube-api-access-gp5mf\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:13.680902 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.680881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zjm\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-kube-api-access-49zjm\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.681090 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.681043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-bound-sa-token\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:13.681291 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.681273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9vrp\" (UniqueName: \"kubernetes.io/projected/5740d722-bab5-4ed8-9085-f5689745880b-kube-api-access-w9vrp\") pod \"ingress-canary-98vk8\" (UID: \"5740d722-bab5-4ed8-9085-f5689745880b\") " pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.749520 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.749484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-98vk8"
Sep 05 02:18:13.770419 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.770389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:13.770578 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.770559    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:13.770644 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.770631    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.770614073 +0000 UTC m=+66.192119393 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:13.871856 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.871817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:13.872039 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:13.871885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:13.872039 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.871995    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:13.872039 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.872007    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:13.872039 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.872015    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:13.872039 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.872027    2582 projected.go:196] Error preparing data for projected volume kube-api-access-4vcbk for pod openshift-network-diagnostics/network-check-target-tvs2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:13.872216 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.872074    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret podName:676f3777-f8f1-4dcc-bf7b-6f95e34c07e2 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.872054605 +0000 UTC m=+66.293559925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret") pod "global-pull-secret-syncer-6rg87" (UID: "676f3777-f8f1-4dcc-bf7b-6f95e34c07e2") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:13.872216 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:13.872093    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk podName:01e674ec-92bd-4c4d-9efc-78b1268bd8e8 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.872083539 +0000 UTC m=+66.293588862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vcbk" (UniqueName: "kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk") pod "network-check-target-tvs2z" (UID: "01e674ec-92bd-4c4d-9efc-78b1268bd8e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:14.174083 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:14.174049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:14.174083 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:14.174096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:14.174498 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:14.174192    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:14.174498 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:14.174206    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:14.174498 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:14.174204    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:14.174498 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:14.174254    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:18:15.174241484 +0000 UTC m=+35.595746801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:14.174498 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:14.174278    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:15.174258958 +0000 UTC m=+35.595764278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:14.478752 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:14.478679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-98vk8"]
Sep 05 02:18:14.482352 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:18:14.482326    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5740d722_bab5_4ed8_9085_f5689745880b.slice/crio-3543ce293c53e27d267de05a4907f9529a9c1b54c7b7676fb8adae119f983275 WatchSource:0}: Error finding container 3543ce293c53e27d267de05a4907f9529a9c1b54c7b7676fb8adae119f983275: Status 404 returned error can't find the container with id 3543ce293c53e27d267de05a4907f9529a9c1b54c7b7676fb8adae119f983275
Sep 05 02:18:15.104249 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.104220    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:15.104436 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.104332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:15.104436 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.104361    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:15.107466 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.107303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:15.107466 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.107306    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:15.107466 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.107339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:15.108393 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.108375    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk4wx\""
Sep 05 02:18:15.108620 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.108599    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4nzpg\""
Sep 05 02:18:15.108731 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.108608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:15.187060 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.187015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:15.187936 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.187076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:15.187936 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:15.187198    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:15.187936 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:15.187219    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:15.187936 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:15.187285    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.187267674 +0000 UTC m=+37.608773007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:15.187936 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:15.187198    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:15.187936 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:15.187360    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.187337615 +0000 UTC m=+37.608842935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:15.289264 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.289227    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5201b1d-866d-4e87-a351-5e7b24756371" containerID="9fb8bf5ec210831fd94dd5fbefba5773f153acd84ce8928caadb8d212235e744" exitCode=0
Sep 05 02:18:15.289413 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.289260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerDied","Data":"9fb8bf5ec210831fd94dd5fbefba5773f153acd84ce8928caadb8d212235e744"}
Sep 05 02:18:15.290771 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:15.290716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-98vk8" event={"ID":"5740d722-bab5-4ed8-9085-f5689745880b","Type":"ContainerStarted","Data":"3543ce293c53e27d267de05a4907f9529a9c1b54c7b7676fb8adae119f983275"}
Sep 05 02:18:16.296926 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:16.296891    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5201b1d-866d-4e87-a351-5e7b24756371" containerID="8d2d1d9566c36f9a9f0239ec9e46ce2dd4b03a88bd4b7f3c045c6f6763036e90" exitCode=0
Sep 05 02:18:16.297377 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:16.296939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerDied","Data":"8d2d1d9566c36f9a9f0239ec9e46ce2dd4b03a88bd4b7f3c045c6f6763036e90"}
Sep 05 02:18:17.201611 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:17.201427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:17.201826 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:17.201645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:17.201826 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:17.201576    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:17.201826 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:17.201736    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:17.201826 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:17.201746    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:17.201826 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:17.201763    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:21.201744132 +0000 UTC m=+41.623249467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:17.201826 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:17.201783    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:18:21.201771862 +0000 UTC m=+41.623277196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:17.306906 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:17.306815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" event={"ID":"c5201b1d-866d-4e87-a351-5e7b24756371","Type":"ContainerStarted","Data":"4675904a36a0fe908c9b56acf4309de1f9999fe151135b5c24e3db7529ab8c69"}
Sep 05 02:18:17.308314 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:17.308285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-98vk8" event={"ID":"5740d722-bab5-4ed8-9085-f5689745880b","Type":"ContainerStarted","Data":"0674a9a0e8131eef78e96724a71f1bd179344298a3a93db4ba4727f55a4af092"}
Sep 05 02:18:17.332582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:17.332531    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4hwq4" podStartSLOduration=9.154448932 podStartE2EDuration="37.332515893s" totalImagesPullingTime="28.178066961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:17.331124017 +0000 UTC m=+37.752629377" watchObservedRunningTime="2026-09-05 02:18:17.332515893 +0000 UTC m=+37.754021232"
Sep 05 02:18:17.349178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:17.349127    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-98vk8" podStartSLOduration=2.108052082 podStartE2EDuration="4.34911022s" totalImagesPullingTime="2.241058138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:17.348178691 +0000 UTC m=+37.769684044" watchObservedRunningTime="2026-09-05 02:18:17.34911022 +0000 UTC m=+37.770615607"
Sep 05 02:18:21.233417 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:21.233372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:21.233805 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:21.233448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:21.233805 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:21.233536    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:21.233805 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:21.233563    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:21.233805 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:21.233539    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:21.233805 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:21.233617    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:18:29.233601538 +0000 UTC m=+49.655106855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:21.233805 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:21.233632    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:29.233626345 +0000 UTC m=+49.655131661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:29.293558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:29.293509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:29.293558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:29.293565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:29.294085 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:29.293670    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:29.294085 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:29.293759    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.293745303 +0000 UTC m=+65.715250620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:29.294085 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:29.293680    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:29.294085 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:29.293780    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:29.294085 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:29.293828    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.293817221 +0000 UTC m=+65.715322538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:39.285862 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:39.285833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njtb5"
Sep 05 02:18:45.301028 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.300987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:18:45.301415 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.301042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:18:45.301415 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.301163    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:45.301415 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.301229    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls podName:f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:17.301211889 +0000 UTC m=+97.722717210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls") pod "dns-default-mgvxp" (UID: "f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:45.301415 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.301166    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:45.301415 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.301272    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c554659c7-k9ljc: secret "image-registry-tls" not found
Sep 05 02:18:45.301415 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.301323    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls podName:38e3cd40-d180-4e9d-9ac9-9b004615c1bf nodeName:}" failed. No retries permitted until 2026-09-05 02:19:17.301311667 +0000 UTC m=+97.722817001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls") pod "image-registry-5c554659c7-k9ljc" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf") : secret "image-registry-tls" not found
Sep 05 02:18:45.804823 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.804777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:18:45.807483 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.807462    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:45.815542 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.815523    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:18:45.815622 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:45.815579    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs podName:9f623e00-8e5e-4eb1-8f53-3751e8474c52 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:49.815563571 +0000 UTC m=+130.237068888 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs") pod "network-metrics-daemon-qkm56" (UID: "9f623e00-8e5e-4eb1-8f53-3751e8474c52") : secret "metrics-daemon-secret" not found
Sep 05 02:18:45.905225 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.905185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:45.905413 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.905247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:45.907911 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.907893    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:45.907991 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.907961    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:45.918021 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.917998    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:45.919065 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.919047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676f3777-f8f1-4dcc-bf7b-6f95e34c07e2-original-pull-secret\") pod \"global-pull-secret-syncer-6rg87\" (UID: \"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2\") " pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:45.928021 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:45.927998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcbk\" (UniqueName: \"kubernetes.io/projected/01e674ec-92bd-4c4d-9efc-78b1268bd8e8-kube-api-access-4vcbk\") pod \"network-check-target-tvs2z\" (UID: \"01e674ec-92bd-4c4d-9efc-78b1268bd8e8\") " pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:46.015801 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.015765    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rg87"
Sep 05 02:18:46.030691 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.030668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hk4wx\""
Sep 05 02:18:46.038861 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.038840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:46.313062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.313033    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6rg87"]
Sep 05 02:18:46.316066 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:18:46.316008    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676f3777_f8f1_4dcc_bf7b_6f95e34c07e2.slice/crio-c3312baa5f661df43767a86ee5350b52b6824643d6a946a312cbece8a4427ebc WatchSource:0}: Error finding container c3312baa5f661df43767a86ee5350b52b6824643d6a946a312cbece8a4427ebc: Status 404 returned error can't find the container with id c3312baa5f661df43767a86ee5350b52b6824643d6a946a312cbece8a4427ebc
Sep 05 02:18:46.336107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.336083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tvs2z"]
Sep 05 02:18:46.339119 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:18:46.339095    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e674ec_92bd_4c4d_9efc_78b1268bd8e8.slice/crio-5160ebdd8659406ed69ee3445da6f3be1807631d46fd090ce8c28ab3e6a5e7a7 WatchSource:0}: Error finding container 5160ebdd8659406ed69ee3445da6f3be1807631d46fd090ce8c28ab3e6a5e7a7: Status 404 returned error can't find the container with id 5160ebdd8659406ed69ee3445da6f3be1807631d46fd090ce8c28ab3e6a5e7a7
Sep 05 02:18:46.381636 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.381593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvs2z" event={"ID":"01e674ec-92bd-4c4d-9efc-78b1268bd8e8","Type":"ContainerStarted","Data":"5160ebdd8659406ed69ee3445da6f3be1807631d46fd090ce8c28ab3e6a5e7a7"}
Sep 05 02:18:46.382696 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:46.382676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6rg87" event={"ID":"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2","Type":"ContainerStarted","Data":"c3312baa5f661df43767a86ee5350b52b6824643d6a946a312cbece8a4427ebc"}
Sep 05 02:18:51.398330 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:51.398289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6rg87" event={"ID":"676f3777-f8f1-4dcc-bf7b-6f95e34c07e2","Type":"ContainerStarted","Data":"8ee0bed8553da30f7affdc261155dc0d980bfa408d634f6a587fc76d2b18c035"}
Sep 05 02:18:51.399712 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:51.399691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tvs2z" event={"ID":"01e674ec-92bd-4c4d-9efc-78b1268bd8e8","Type":"ContainerStarted","Data":"7674fcbcae16074164a109d06ed99aca5679dba12d8b1a3e7b1aabd941a3bd3b"}
Sep 05 02:18:51.399844 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:51.399832    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:18:51.423079 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:51.423028    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6rg87" podStartSLOduration=66.715651375 podStartE2EDuration="1m11.423012568s" totalImagesPullingTime="4.707361193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:51.421825986 +0000 UTC m=+71.843331326" watchObservedRunningTime="2026-09-05 02:18:51.423012568 +0000 UTC m=+71.844517906"
Sep 05 02:18:51.441157 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:51.441109    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tvs2z" podStartSLOduration=66.760674376 podStartE2EDuration="1m11.44109416s" totalImagesPullingTime="4.680419784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:51.44043736 +0000 UTC m=+71.861942693" watchObservedRunningTime="2026-09-05 02:18:51.44109416 +0000 UTC m=+71.862599499"
Sep 05 02:18:52.863104 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.863069    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-797dcfd87b-8dqvp"]
Sep 05 02:18:52.865176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.865153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:52.870173 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.870149    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xtdmn\""
Sep 05 02:18:52.870286 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.870177    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 05 02:18:52.870286 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.870259    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 05 02:18:52.870458 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.870442    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 05 02:18:52.870501 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.870445    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 02:18:52.871453 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.871149    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 05 02:18:52.871453 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.871184    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 02:18:52.881078 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.881051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-797dcfd87b-8dqvp"]
Sep 05 02:18:52.968940 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.968886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59qq\" (UniqueName: \"kubernetes.io/projected/d3522c0a-02bd-42f1-8931-941e41b9151e-kube-api-access-c59qq\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:52.968940 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.968948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:52.969178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.969034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-default-certificate\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:52.969178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.969070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-stats-auth\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:52.969178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:52.969127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.069599 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.069556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.069599 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.069604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59qq\" (UniqueName: \"kubernetes.io/projected/d3522c0a-02bd-42f1-8931-941e41b9151e-kube-api-access-c59qq\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.069851 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.069627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.069851 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.069650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-default-certificate\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.069851 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.069676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-stats-auth\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.069851 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:53.069738    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:53.069851 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:53.069762    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.569740933 +0000 UTC m=+73.991246249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:53.069851 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:53.069805    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.569785401 +0000 UTC m=+73.991290733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : secret "router-metrics-certs-default" not found
Sep 05 02:18:53.072192 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.072174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-stats-auth\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.072250 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.072180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-default-certificate\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.078309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.078289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59qq\" (UniqueName: \"kubernetes.io/projected/d3522c0a-02bd-42f1-8931-941e41b9151e-kube-api-access-c59qq\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.573442 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.573401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.573604 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:53.573465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:53.573646 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:53.573601    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:53.573646 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:53.573603    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.573584602 +0000 UTC m=+74.995089934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:53.573761 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:53.573655    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.573642773 +0000 UTC m=+74.995148093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : secret "router-metrics-certs-default" not found
Sep 05 02:18:54.580424 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:54.580372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:54.580424 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:54.580434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:54.580891 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:54.580539    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:56.580526282 +0000 UTC m=+77.002031599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:54.580891 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:54.580540    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:54.580891 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:54.580612    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:56.580593026 +0000 UTC m=+77.002098361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : secret "router-metrics-certs-default" not found
Sep 05 02:18:56.593985 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:56.593929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:56.594377 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:56.594000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:18:56.594377 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:56.594069    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:18:56.594377 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:56.594120    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.594106834 +0000 UTC m=+81.015612151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : secret "router-metrics-certs-default" not found
Sep 05 02:18:56.594377 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:56.594134    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.594128327 +0000 UTC m=+81.015633643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : configmap references non-existent config key: service-ca.crt
Sep 05 02:18:57.606484 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.606450    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xvtld"]
Sep 05 02:18:57.608390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.608373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.611691 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.611673    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:18:57.611993 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.611957    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:18:57.612063 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.611998    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:18:57.612659 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.612640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:18:57.615950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.615932    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7x6sm\""
Sep 05 02:18:57.635092 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.635064    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvtld"]
Sep 05 02:18:57.700437 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.700400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e549ee2-39c0-4387-b159-c7a58c93f6d3-crio-socket\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.700598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.700464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.700598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.700511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e549ee2-39c0-4387-b159-c7a58c93f6d3-data-volume\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.700598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.700552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.700598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.700568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cn2t\" (UniqueName: \"kubernetes.io/projected/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-api-access-6cn2t\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801195 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e549ee2-39c0-4387-b159-c7a58c93f6d3-crio-socket\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801355 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801355 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e549ee2-39c0-4387-b159-c7a58c93f6d3-data-volume\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801355 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e549ee2-39c0-4387-b159-c7a58c93f6d3-crio-socket\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801451 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801451 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:57.801384    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:57.801451 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:57.801427    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:58.301415669 +0000 UTC m=+78.722920986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:57.801451 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cn2t\" (UniqueName: \"kubernetes.io/projected/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-api-access-6cn2t\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.801596 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:57.801518    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:57.801596 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:57.801577    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:58.301561161 +0000 UTC m=+78.723066489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:57.801674 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.801592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e549ee2-39c0-4387-b159-c7a58c93f6d3-data-volume\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:57.818789 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:57.818764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cn2t\" (UniqueName: \"kubernetes.io/projected/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-api-access-6cn2t\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:58.305373 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:58.305311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:58.305563 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:58.305423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:58.305563 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:58.305452    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:58.305563 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:58.305523    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:58.305563 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:58.305525    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.305509547 +0000 UTC m=+79.727014864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:58.305717 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:58.305572    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.305560733 +0000 UTC m=+79.727066050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.313278 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:59.313239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:59.313665 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:59.313301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:18:59.313665 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:59.313378    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.313665 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:59.313442    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:01.313426066 +0000 UTC m=+81.734931383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.313665 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:59.313387    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.313665 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:18:59.313526    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:01.313514343 +0000 UTC m=+81.735019660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.369369 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:18:59.369336    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42lzd_2e7f06bf-d675-415e-8a76-1ec169531719/dns-node-resolver/0.log"
Sep 05 02:19:00.622763 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:00.622710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:00.623184 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:00.622781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:00.623184 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:00.622863    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 02:19:00.623184 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:00.622899    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:08.622886079 +0000 UTC m=+89.044391395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : configmap references non-existent config key: service-ca.crt
Sep 05 02:19:00.623184 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:00.622920    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs podName:d3522c0a-02bd-42f1-8931-941e41b9151e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:08.622907092 +0000 UTC m=+89.044412408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs") pod "router-default-797dcfd87b-8dqvp" (UID: "d3522c0a-02bd-42f1-8931-941e41b9151e") : secret "router-metrics-certs-default" not found
Sep 05 02:19:00.770233 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:00.770205    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhnn2_bd894870-c9ae-4904-a92f-6edba4702fbc/node-ca/0.log"
Sep 05 02:19:01.170563 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:01.170532    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-98vk8_5740d722-bab5-4ed8-9085-f5689745880b/serve-healthcheck-canary/0.log"
Sep 05 02:19:01.328343 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:01.328302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:01.328520 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:01.328401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:01.328520 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:01.328421    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:01.328520 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:01.328482    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.328466867 +0000 UTC m=+85.749972185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:01.328520 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:01.328502    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:01.328656 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:01.328549    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:05.328531554 +0000 UTC m=+85.750036871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:05.360013 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:05.359846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:05.360013 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:05.359924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:05.360492 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:05.360020    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:05.360492 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:05.360051    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:19:05.360492 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:05.360086    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:13.360071622 +0000 UTC m=+93.781576939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:19:05.360492 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:19:05.360121    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm podName:0e549ee2-39c0-4387-b159-c7a58c93f6d3 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:13.36010498 +0000 UTC m=+93.781610320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvtld" (UID: "0e549ee2-39c0-4387-b159-c7a58c93f6d3") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:08.683350 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:08.683313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:08.683728 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:08.683382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:08.684071 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:08.684051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3522c0a-02bd-42f1-8931-941e41b9151e-service-ca-bundle\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:08.685664 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:08.685646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3522c0a-02bd-42f1-8931-941e41b9151e-metrics-certs\") pod \"router-default-797dcfd87b-8dqvp\" (UID: \"d3522c0a-02bd-42f1-8931-941e41b9151e\") " pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:08.773924 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:08.773885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:09.055089 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:09.055004    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-797dcfd87b-8dqvp"]
Sep 05 02:19:09.059180 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:09.059152    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3522c0a_02bd_42f1_8931_941e41b9151e.slice/crio-03ce65fe3e2375d2dd9b18904bad4ef28e10dc8d228b74c1eb0e39a256f50f86 WatchSource:0}: Error finding container 03ce65fe3e2375d2dd9b18904bad4ef28e10dc8d228b74c1eb0e39a256f50f86: Status 404 returned error can't find the container with id 03ce65fe3e2375d2dd9b18904bad4ef28e10dc8d228b74c1eb0e39a256f50f86
Sep 05 02:19:09.443207 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:09.443172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-797dcfd87b-8dqvp" event={"ID":"d3522c0a-02bd-42f1-8931-941e41b9151e","Type":"ContainerStarted","Data":"75bb579fd68e1bde4053d3cb5be4cf06b18545a47512d9f5286ae483451c6799"}
Sep 05 02:19:09.443450 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:09.443227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-797dcfd87b-8dqvp" event={"ID":"d3522c0a-02bd-42f1-8931-941e41b9151e","Type":"ContainerStarted","Data":"03ce65fe3e2375d2dd9b18904bad4ef28e10dc8d228b74c1eb0e39a256f50f86"}
Sep 05 02:19:09.463295 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:09.463242    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-797dcfd87b-8dqvp" podStartSLOduration=17.463226866 podStartE2EDuration="17.463226866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:09.462810565 +0000 UTC m=+89.884315905" watchObservedRunningTime="2026-09-05 02:19:09.463226866 +0000 UTC m=+89.884732204"
Sep 05 02:19:09.774684 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:09.774591    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:09.777257 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:09.777234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:10.445302 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:10.445272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:10.446640 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:10.446618    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-797dcfd87b-8dqvp"
Sep 05 02:19:13.421104 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:13.421065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:13.421499 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:13.421122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:13.421659 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:13.421638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e549ee2-39c0-4387-b159-c7a58c93f6d3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:13.423297 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:13.423278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e549ee2-39c0-4387-b159-c7a58c93f6d3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvtld\" (UID: \"0e549ee2-39c0-4387-b159-c7a58c93f6d3\") " pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:13.517628 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:13.517575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvtld"
Sep 05 02:19:13.793926 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:13.793827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvtld"]
Sep 05 02:19:13.796609 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:13.796579    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e549ee2_39c0_4387_b159_c7a58c93f6d3.slice/crio-d4009cda42224725418037bea67ab8ccd414f388f5e4059fa45f86d17e5c6de3 WatchSource:0}: Error finding container d4009cda42224725418037bea67ab8ccd414f388f5e4059fa45f86d17e5c6de3: Status 404 returned error can't find the container with id d4009cda42224725418037bea67ab8ccd414f388f5e4059fa45f86d17e5c6de3
Sep 05 02:19:14.457458 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:14.457433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvtld" event={"ID":"0e549ee2-39c0-4387-b159-c7a58c93f6d3","Type":"ContainerStarted","Data":"0485c54b3c6be6ba951594bb03493068468ab43ceb114815b3e6e51596e1c09c"}
Sep 05 02:19:14.457745 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:14.457484    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvtld" event={"ID":"0e549ee2-39c0-4387-b159-c7a58c93f6d3","Type":"ContainerStarted","Data":"f419f817c38280420106dffbc16a81ae8a3ebf04b19a1610633266c6fe23ccf9"}
Sep 05 02:19:14.457745 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:14.457503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvtld" event={"ID":"0e549ee2-39c0-4387-b159-c7a58c93f6d3","Type":"ContainerStarted","Data":"d4009cda42224725418037bea67ab8ccd414f388f5e4059fa45f86d17e5c6de3"}
Sep 05 02:19:16.467692 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:16.467649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvtld" event={"ID":"0e549ee2-39c0-4387-b159-c7a58c93f6d3","Type":"ContainerStarted","Data":"9c231dea908eb483f1e77fee08dce5cbf365eead137f3fa2a9febd7f203c3eab"}
Sep 05 02:19:16.488278 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:16.488228    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xvtld" podStartSLOduration=17.80897677 podStartE2EDuration="19.488212757s" totalImagesPullingTime="1.679235987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:16.486701813 +0000 UTC m=+96.908207174" watchObservedRunningTime="2026-09-05 02:19:16.488212757 +0000 UTC m=+96.909718095"
Sep 05 02:19:17.350082 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.350049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:19:17.350252 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.350105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:19:17.352546 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.352510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5-metrics-tls\") pod \"dns-default-mgvxp\" (UID: \"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5\") " pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:19:17.352652 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.352551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"image-registry-5c554659c7-k9ljc\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") " pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:19:17.631621 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.631538    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f79x9\""
Sep 05 02:19:17.638812 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.638795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:19:17.643808 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.643789    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cl7gt\""
Sep 05 02:19:17.652010 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.651984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:19:17.916856 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.916784    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c554659c7-k9ljc"]
Sep 05 02:19:17.920399 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:17.920373    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e3cd40_d180_4e9d_9ac9_9b004615c1bf.slice/crio-a6b0ea2120068dca0d23add09d32b8b9b4a32118cc17760022c0aae4ade70f60 WatchSource:0}: Error finding container a6b0ea2120068dca0d23add09d32b8b9b4a32118cc17760022c0aae4ade70f60: Status 404 returned error can't find the container with id a6b0ea2120068dca0d23add09d32b8b9b4a32118cc17760022c0aae4ade70f60
Sep 05 02:19:17.932857 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:17.932834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mgvxp"]
Sep 05 02:19:17.935698 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:17.935672    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2beaaa7_4e78_4f87_85ba_a22bb1adfcb5.slice/crio-95c692dd005fb9d493d151f3c45438666465f8f3c8ea6184f393ebd142e807e2 WatchSource:0}: Error finding container 95c692dd005fb9d493d151f3c45438666465f8f3c8ea6184f393ebd142e807e2: Status 404 returned error can't find the container with id 95c692dd005fb9d493d151f3c45438666465f8f3c8ea6184f393ebd142e807e2
Sep 05 02:19:18.476203 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:18.476170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" event={"ID":"38e3cd40-d180-4e9d-9ac9-9b004615c1bf","Type":"ContainerStarted","Data":"345356cb43b02ead9dcfc7836771aca5c60aaeda090014fb3874a435b09c98a5"}
Sep 05 02:19:18.476406 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:18.476229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" event={"ID":"38e3cd40-d180-4e9d-9ac9-9b004615c1bf","Type":"ContainerStarted","Data":"a6b0ea2120068dca0d23add09d32b8b9b4a32118cc17760022c0aae4ade70f60"}
Sep 05 02:19:18.476406 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:18.476256    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:19:18.477656 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:18.477631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgvxp" event={"ID":"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5","Type":"ContainerStarted","Data":"95c692dd005fb9d493d151f3c45438666465f8f3c8ea6184f393ebd142e807e2"}
Sep 05 02:19:18.500225 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:18.500138    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" podStartSLOduration=98.500116336 podStartE2EDuration="1m38.500116336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:18.498367067 +0000 UTC m=+98.919872430" watchObservedRunningTime="2026-09-05 02:19:18.500116336 +0000 UTC m=+98.921621677"
Sep 05 02:19:19.484877 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:19.484838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgvxp" event={"ID":"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5","Type":"ContainerStarted","Data":"a880674c95ad38853f2b05213a48c28dbf9567b71d8d685cb564183b356dcc84"}
Sep 05 02:19:20.489612 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:20.489576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mgvxp" event={"ID":"f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5","Type":"ContainerStarted","Data":"3b95bf6ed286204fb0f6a2a61fce5407e284ada81b5ca87ad2b9f3af77f0c435"}
Sep 05 02:19:20.490028 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:20.489711    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:19:20.507440 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:20.507390    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mgvxp" podStartSLOduration=66.095556351 podStartE2EDuration="1m7.507374504s" totalImagesPullingTime="1.411818153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:20.506285159 +0000 UTC m=+100.927790500" watchObservedRunningTime="2026-09-05 02:19:20.507374504 +0000 UTC m=+100.928879850"
Sep 05 02:19:22.404649 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.404619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tvs2z"
Sep 05 02:19:22.668107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.668016    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-w75fs"]
Sep 05 02:19:22.671128 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.671105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.674161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.674141    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 05 02:19:22.674506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.674481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 05 02:19:22.675210 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.675193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kftp9\""
Sep 05 02:19:22.682717 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.682696    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-w75fs"]
Sep 05 02:19:22.685555 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.685526    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"]
Sep 05 02:19:22.688330 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.688310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.690500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.690477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:19:22.690589 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.690525    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:19:22.690766 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.690747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:19:22.691037 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.691021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 05 02:19:22.704024 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.704000    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"]
Sep 05 02:19:22.790219 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.790181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1887f748-d63e-440f-8097-21b65d22c227-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-w75fs\" (UID: \"1887f748-d63e-440f-8097-21b65d22c227\") " pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.790381 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.790230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/de17f961-e9ae-4fa1-b044-16b5fb2acf79-klusterlet-config\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.790381 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.790260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1887f748-d63e-440f-8097-21b65d22c227-nginx-conf\") pod \"networking-console-plugin-79977f9844-w75fs\" (UID: \"1887f748-d63e-440f-8097-21b65d22c227\") " pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.790469 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.790373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/de17f961-e9ae-4fa1-b044-16b5fb2acf79-kube-api-access-tkrgj\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.790469 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.790410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de17f961-e9ae-4fa1-b044-16b5fb2acf79-tmp\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.890944 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.890907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/de17f961-e9ae-4fa1-b044-16b5fb2acf79-kube-api-access-tkrgj\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.890944 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.890949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de17f961-e9ae-4fa1-b044-16b5fb2acf79-tmp\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.891193 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.891085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1887f748-d63e-440f-8097-21b65d22c227-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-w75fs\" (UID: \"1887f748-d63e-440f-8097-21b65d22c227\") " pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.891193 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.891103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/de17f961-e9ae-4fa1-b044-16b5fb2acf79-klusterlet-config\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.891193 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.891120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1887f748-d63e-440f-8097-21b65d22c227-nginx-conf\") pod \"networking-console-plugin-79977f9844-w75fs\" (UID: \"1887f748-d63e-440f-8097-21b65d22c227\") " pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.891496 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.891474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de17f961-e9ae-4fa1-b044-16b5fb2acf79-tmp\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.891880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.891860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1887f748-d63e-440f-8097-21b65d22c227-nginx-conf\") pod \"networking-console-plugin-79977f9844-w75fs\" (UID: \"1887f748-d63e-440f-8097-21b65d22c227\") " pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.893471 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.893449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1887f748-d63e-440f-8097-21b65d22c227-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-w75fs\" (UID: \"1887f748-d63e-440f-8097-21b65d22c227\") " pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.893561 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.893551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/de17f961-e9ae-4fa1-b044-16b5fb2acf79-klusterlet-config\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.903236 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.903212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrgj\" (UniqueName: \"kubernetes.io/projected/de17f961-e9ae-4fa1-b044-16b5fb2acf79-kube-api-access-tkrgj\") pod \"klusterlet-addon-workmgr-58474fd9f8-2gmxr\" (UID: \"de17f961-e9ae-4fa1-b044-16b5fb2acf79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:22.979505 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.979421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs"
Sep 05 02:19:22.996787 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:22.996764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:23.250428 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:23.250350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-w75fs"]
Sep 05 02:19:23.253424 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:23.253391    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1887f748_d63e_440f_8097_21b65d22c227.slice/crio-facb0d6b55b858a11cfdb6cca10da7afe68f0e1b0b91891891442cfab0017375 WatchSource:0}: Error finding container facb0d6b55b858a11cfdb6cca10da7afe68f0e1b0b91891891442cfab0017375: Status 404 returned error can't find the container with id facb0d6b55b858a11cfdb6cca10da7afe68f0e1b0b91891891442cfab0017375
Sep 05 02:19:23.277181 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:23.277154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"]
Sep 05 02:19:23.280008 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:23.279980    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde17f961_e9ae_4fa1_b044_16b5fb2acf79.slice/crio-c4d67d3b89093774b01a1b6a1d2d56efab0911d03310e34f08718ba26a8f0aff WatchSource:0}: Error finding container c4d67d3b89093774b01a1b6a1d2d56efab0911d03310e34f08718ba26a8f0aff: Status 404 returned error can't find the container with id c4d67d3b89093774b01a1b6a1d2d56efab0911d03310e34f08718ba26a8f0aff
Sep 05 02:19:23.500632 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:23.500538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs" event={"ID":"1887f748-d63e-440f-8097-21b65d22c227","Type":"ContainerStarted","Data":"facb0d6b55b858a11cfdb6cca10da7afe68f0e1b0b91891891442cfab0017375"}
Sep 05 02:19:23.501583 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:23.501557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr" event={"ID":"de17f961-e9ae-4fa1-b044-16b5fb2acf79","Type":"ContainerStarted","Data":"c4d67d3b89093774b01a1b6a1d2d56efab0911d03310e34f08718ba26a8f0aff"}
Sep 05 02:19:25.510931 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:25.510888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs" event={"ID":"1887f748-d63e-440f-8097-21b65d22c227","Type":"ContainerStarted","Data":"88b7c65d3b9ebbbb5e50ad4dedc4b0049b7a983fb74b4f7954d84bb869b46788"}
Sep 05 02:19:25.530199 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:25.530148    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-w75fs" podStartSLOduration=2.283212766 podStartE2EDuration="3.530127427s" totalImagesPullingTime="1.246914661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:25.529250705 +0000 UTC m=+105.950756043" watchObservedRunningTime="2026-09-05 02:19:25.530127427 +0000 UTC m=+105.951632766"
Sep 05 02:19:27.518596 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:27.518557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr" event={"ID":"de17f961-e9ae-4fa1-b044-16b5fb2acf79","Type":"ContainerStarted","Data":"e6e241d1daba3fdec0b3debd14b7d3b29b5cc4a52893c06ddf7ecae759bb4217"}
Sep 05 02:19:27.519015 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:27.518781    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:27.520326 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:27.520307    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr"
Sep 05 02:19:27.535474 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:27.535425    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58474fd9f8-2gmxr" podStartSLOduration=1.469404338 podStartE2EDuration="5.535410062s" totalImagesPullingTime="4.066005724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:27.533646569 +0000 UTC m=+107.955151908" watchObservedRunningTime="2026-09-05 02:19:27.535410062 +0000 UTC m=+107.956915394"
Sep 05 02:19:30.496032 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:30.495999    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mgvxp"
Sep 05 02:19:32.164622 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.164586    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"]
Sep 05 02:19:32.166975 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.166950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.171628 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.171599    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 05 02:19:32.171876 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.171851    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:19:32.172027 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.171904    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:19:32.172027 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.171944    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:19:32.172713 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.172691    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wsld8\""
Sep 05 02:19:32.172840 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.172724    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:19:32.199937 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.199905    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"]
Sep 05 02:19:32.241031 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.240993    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"]
Sep 05 02:19:32.243485 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.243458    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.246911 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.246890    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 05 02:19:32.247069 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.246930    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:19:32.247069 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.247059    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kttvr\""
Sep 05 02:19:32.256185 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.256162    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vtpdm"]
Sep 05 02:19:32.258520 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.258502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.262630 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.262578    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"]
Sep 05 02:19:32.267808 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.267787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.267915 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.267820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.267915 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.267898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r4w\" (UniqueName: \"kubernetes.io/projected/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-kube-api-access-k9r4w\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.268050 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.267993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.271319 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.271303    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 05 02:19:32.273004 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.272986    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:19:32.273004 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.273000    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pzst5\""
Sep 05 02:19:32.273155 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.273082    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:19:32.273155 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.273082    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:19:32.368553 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-sys\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.368553 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-textfile\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.368756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-root\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.368756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.368756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-wtmp\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.368756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2b5b2a6-f203-4cfb-981f-6d840d74173e-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.368756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.368917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-tls\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.368917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.368917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2b5b2a6-f203-4cfb-981f-6d840d74173e-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.368917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.368917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/553f165a-4f84-41e5-bc8f-5007cccfcd97-metrics-client-ca\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.368917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.368897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggxr\" (UniqueName: \"kubernetes.io/projected/553f165a-4f84-41e5-bc8f-5007cccfcd97-kube-api-access-7ggxr\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.369171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.369171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.369171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r4w\" (UniqueName: \"kubernetes.io/projected/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-kube-api-access-k9r4w\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.369171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.369349 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pxx\" (UniqueName: \"kubernetes.io/projected/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-api-access-f2pxx\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.369349 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.369592 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.369576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.371195 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.371175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.371334 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.371317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.403335 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.403291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r4w\" (UniqueName: \"kubernetes.io/projected/fc51dc11-4d1b-469c-b3b9-7487b5e2861d-kube-api-access-k9r4w\") pod \"openshift-state-metrics-657fd6c54-jdpgg\" (UID: \"fc51dc11-4d1b-469c-b3b9-7487b5e2861d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pxx\" (UniqueName: \"kubernetes.io/projected/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-api-access-f2pxx\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-sys\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-textfile\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-root\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-wtmp\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2b5b2a6-f203-4cfb-981f-6d840d74173e-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-tls\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.470984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2b5b2a6-f203-4cfb-981f-6d840d74173e-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/553f165a-4f84-41e5-bc8f-5007cccfcd97-metrics-client-ca\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggxr\" (UniqueName: \"kubernetes.io/projected/553f165a-4f84-41e5-bc8f-5007cccfcd97-kube-api-access-7ggxr\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.474239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-accelerators-collector-config\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.471754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-sys\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.472475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-root\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.473300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2b5b2a6-f203-4cfb-981f-6d840d74173e-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.473455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-wtmp\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.474155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.474467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-tls\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.474736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/553f165a-4f84-41e5-bc8f-5007cccfcd97-metrics-client-ca\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475105 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.475033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"
Sep 05 02:19:32.475589 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.475117    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2b5b2a6-f203-4cfb-981f-6d840d74173e-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.475589 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.475179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-textfile\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.475589 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.475565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/553f165a-4f84-41e5-bc8f-5007cccfcd97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.477043 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.477016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.477329 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.477306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.483060 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.483034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggxr\" (UniqueName: \"kubernetes.io/projected/553f165a-4f84-41e5-bc8f-5007cccfcd97-kube-api-access-7ggxr\") pod \"node-exporter-vtpdm\" (UID: \"553f165a-4f84-41e5-bc8f-5007cccfcd97\") " pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.483318 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.483300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pxx\" (UniqueName: \"kubernetes.io/projected/e2b5b2a6-f203-4cfb-981f-6d840d74173e-kube-api-access-f2pxx\") pod \"kube-state-metrics-74cb57df7-ljgvf\" (UID: \"e2b5b2a6-f203-4cfb-981f-6d840d74173e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.552507 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.552479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"
Sep 05 02:19:32.567865 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.567841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vtpdm"
Sep 05 02:19:32.577411 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:32.577378    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553f165a_4f84_41e5_bc8f_5007cccfcd97.slice/crio-9905807c1a14d14707bbe1deaf33e0cb01b5b1d755f8184f75f8bc24c42cdc9c WatchSource:0}: Error finding container 9905807c1a14d14707bbe1deaf33e0cb01b5b1d755f8184f75f8bc24c42cdc9c: Status 404 returned error can't find the container with id 9905807c1a14d14707bbe1deaf33e0cb01b5b1d755f8184f75f8bc24c42cdc9c
Sep 05 02:19:32.747855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.747770    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg"]
Sep 05 02:19:32.750610 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:32.750587    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc51dc11_4d1b_469c_b3b9_7487b5e2861d.slice/crio-19053080a0cf418374bef7d4dc81b7d5cd4a37a41507d195c388af8540319564 WatchSource:0}: Error finding container 19053080a0cf418374bef7d4dc81b7d5cd4a37a41507d195c388af8540319564: Status 404 returned error can't find the container with id 19053080a0cf418374bef7d4dc81b7d5cd4a37a41507d195c388af8540319564
Sep 05 02:19:32.835060 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.835031    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf"]
Sep 05 02:19:32.839083 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:32.839048    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b5b2a6_f203_4cfb_981f_6d840d74173e.slice/crio-f21954da655c8e1489cc38fb43b5a2b923c3dff87d739fda3c0c779427263dec WatchSource:0}: Error finding container f21954da655c8e1489cc38fb43b5a2b923c3dff87d739fda3c0c779427263dec: Status 404 returned error can't find the container with id f21954da655c8e1489cc38fb43b5a2b923c3dff87d739fda3c0c779427263dec
Sep 05 02:19:32.975429 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.975394    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:19:32.978847 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.978830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:32.983505 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.983482    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 02:19:32.984242 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984214    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 02:19:32.984242 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984228    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 02:19:32.984242 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984230    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 02:19:32.984462 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984245    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 02:19:32.984462 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984216    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 02:19:32.984596 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984581    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 02:19:32.984951 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.984935    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-szr26\""
Sep 05 02:19:32.985707 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.985689    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 02:19:32.985795 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.985692    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 02:19:32.994204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:32.994181    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:19:33.075088 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-config-out\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075088 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-web-config\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5sj\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-kube-api-access-wd5sj\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075790 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-config-volume\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075790 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075790 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.075790 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.075540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-web-config\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5sj\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-kube-api-access-wd5sj\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-config-volume\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-config-out\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.176672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.176493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.177240 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.177063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.177320 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.177294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.177379 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.177357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.179769 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.179739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-web-config\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.179900 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.179796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-config-out\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.179900 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.179868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-config-volume\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.180008 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.179937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.180089 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.180070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.180141 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.180125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.180187 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.180167    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.180242 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.180229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.180998 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.180959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.190032 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.190004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5sj\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-kube-api-access-wd5sj\") pod \"alertmanager-main-0\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.288114 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.288079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:19:33.542501 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.542409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpdm" event={"ID":"553f165a-4f84-41e5-bc8f-5007cccfcd97","Type":"ContainerStarted","Data":"9905807c1a14d14707bbe1deaf33e0cb01b5b1d755f8184f75f8bc24c42cdc9c"}
Sep 05 02:19:33.544392 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.544317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf" event={"ID":"e2b5b2a6-f203-4cfb-981f-6d840d74173e","Type":"ContainerStarted","Data":"f21954da655c8e1489cc38fb43b5a2b923c3dff87d739fda3c0c779427263dec"}
Sep 05 02:19:33.547410 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.547021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg" event={"ID":"fc51dc11-4d1b-469c-b3b9-7487b5e2861d","Type":"ContainerStarted","Data":"ba62535d7841c04247e4e454ca6ad451e1eee92cc6026f506f440dcd20e5c9f8"}
Sep 05 02:19:33.547410 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.547076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg" event={"ID":"fc51dc11-4d1b-469c-b3b9-7487b5e2861d","Type":"ContainerStarted","Data":"9f5aa4101bee00a2bd18b12b70e3ae133c1eff1141ad485d3931f79368b93dc0"}
Sep 05 02:19:33.547410 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.547106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg" event={"ID":"fc51dc11-4d1b-469c-b3b9-7487b5e2861d","Type":"ContainerStarted","Data":"19053080a0cf418374bef7d4dc81b7d5cd4a37a41507d195c388af8540319564"}
Sep 05 02:19:33.579373 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:33.578946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:19:33.763294 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:33.763225    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e10383_4fba_4499_8a0b_ce741fe41e71.slice/crio-236ec73ce8a9456c686ae0c0829d5317cf45bd735aa1f99278f97e713adbdf4d WatchSource:0}: Error finding container 236ec73ce8a9456c686ae0c0829d5317cf45bd735aa1f99278f97e713adbdf4d: Status 404 returned error can't find the container with id 236ec73ce8a9456c686ae0c0829d5317cf45bd735aa1f99278f97e713adbdf4d
Sep 05 02:19:34.562305 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.561801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"236ec73ce8a9456c686ae0c0829d5317cf45bd735aa1f99278f97e713adbdf4d"}
Sep 05 02:19:34.565642 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.565607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpdm" event={"ID":"553f165a-4f84-41e5-bc8f-5007cccfcd97","Type":"ContainerStarted","Data":"95c55fa3ce2c5a5a13aca4914779a4b66e570b6429ddb42f1499114706f5aeb1"}
Sep 05 02:19:34.569711 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.569678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf" event={"ID":"e2b5b2a6-f203-4cfb-981f-6d840d74173e","Type":"ContainerStarted","Data":"6f23b1d378d1fd686eb53fa4d3be2344427c3d5813a310643f7d832c30a3dbb7"}
Sep 05 02:19:34.569824 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.569744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf" event={"ID":"e2b5b2a6-f203-4cfb-981f-6d840d74173e","Type":"ContainerStarted","Data":"85786736613a3b7191ba020333e1757f6c18dd7de864b6468e6c9f14e0d65bc5"}
Sep 05 02:19:34.574982 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.574895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg" event={"ID":"fc51dc11-4d1b-469c-b3b9-7487b5e2861d","Type":"ContainerStarted","Data":"b66f2cacc49c7f5400a0e7a4eb8de249332f949c945f8820801bc3a2dcfb9462"}
Sep 05 02:19:34.968596 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.968486    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jdpgg" podStartSLOduration=1.493368624 podStartE2EDuration="2.968466837s" totalImagesPullingTime="1.475098213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:34.60705068 +0000 UTC m=+115.028556021" watchObservedRunningTime="2026-09-05 02:19:34.968466837 +0000 UTC m=+115.389972176"
Sep 05 02:19:34.969159 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.969128    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"]
Sep 05 02:19:34.973420 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.973392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:34.975878 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.975851    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 05 02:19:34.976347 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.976326    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rdxz8\""
Sep 05 02:19:34.976454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.976377    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 05 02:19:34.976454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.976421    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 05 02:19:34.977399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.977379    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 05 02:19:34.977511 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.977443    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-arm27fso90gmd\""
Sep 05 02:19:34.977705 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.977683    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 05 02:19:34.987066 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:34.987040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"]
Sep 05 02:19:35.092874 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.092844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-grpc-tls\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093036 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.092888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093036 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.092950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8f9d653-c15b-4d51-bbde-2b05b30fef65-metrics-client-ca\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093036 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.092991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093036 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.093015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vc4\" (UniqueName: \"kubernetes.io/projected/a8f9d653-c15b-4d51-bbde-2b05b30fef65-kube-api-access-j8vc4\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093174 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.093041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093174 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.093116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-tls\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.093174 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.093142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194413 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-grpc-tls\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194565 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194565 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8f9d653-c15b-4d51-bbde-2b05b30fef65-metrics-client-ca\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194565 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194565 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vc4\" (UniqueName: \"kubernetes.io/projected/a8f9d653-c15b-4d51-bbde-2b05b30fef65-kube-api-access-j8vc4\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194565 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194816 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-tls\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.194816 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.194619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.195409 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.195356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8f9d653-c15b-4d51-bbde-2b05b30fef65-metrics-client-ca\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.197428 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.197373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-tls\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.197541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.197444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.197541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.197519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-grpc-tls\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.198087 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.197877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.198087 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.198046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.198205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.198086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a8f9d653-c15b-4d51-bbde-2b05b30fef65-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.203175 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.203157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vc4\" (UniqueName: \"kubernetes.io/projected/a8f9d653-c15b-4d51-bbde-2b05b30fef65-kube-api-access-j8vc4\") pod \"thanos-querier-db4fbb5df-gwzz2\" (UID: \"a8f9d653-c15b-4d51-bbde-2b05b30fef65\") " pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.284337 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.284240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:35.558166 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.558089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"]
Sep 05 02:19:35.561181 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:35.561154    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f9d653_c15b_4d51_bbde_2b05b30fef65.slice/crio-40a227ddcdafda0af8d0e2ca15663f36ab9f8570e8d338dcc5bfefe321104b1a WatchSource:0}: Error finding container 40a227ddcdafda0af8d0e2ca15663f36ab9f8570e8d338dcc5bfefe321104b1a: Status 404 returned error can't find the container with id 40a227ddcdafda0af8d0e2ca15663f36ab9f8570e8d338dcc5bfefe321104b1a
Sep 05 02:19:35.579582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.579553    2582 generic.go:356] "Generic (PLEG): container finished" podID="553f165a-4f84-41e5-bc8f-5007cccfcd97" containerID="95c55fa3ce2c5a5a13aca4914779a4b66e570b6429ddb42f1499114706f5aeb1" exitCode=0
Sep 05 02:19:35.579930 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.579639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpdm" event={"ID":"553f165a-4f84-41e5-bc8f-5007cccfcd97","Type":"ContainerDied","Data":"95c55fa3ce2c5a5a13aca4914779a4b66e570b6429ddb42f1499114706f5aeb1"}
Sep 05 02:19:35.581917 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.581891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf" event={"ID":"e2b5b2a6-f203-4cfb-981f-6d840d74173e","Type":"ContainerStarted","Data":"b01a812d30e70ff3541043297150458e51fe21f4b87c5b3f548803b4b6775616"}
Sep 05 02:19:35.583519 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.583484    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0" exitCode=0
Sep 05 02:19:35.583597 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.583526    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"}
Sep 05 02:19:35.584855 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.584836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"40a227ddcdafda0af8d0e2ca15663f36ab9f8570e8d338dcc5bfefe321104b1a"}
Sep 05 02:19:35.622086 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:35.622039    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-ljgvf" podStartSLOduration=2.112157676 podStartE2EDuration="3.622022578s" totalImagesPullingTime="1.509864902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:35.620953362 +0000 UTC m=+116.042458702" watchObservedRunningTime="2026-09-05 02:19:35.622022578 +0000 UTC m=+116.043527923"
Sep 05 02:19:36.594189 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.594140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpdm" event={"ID":"553f165a-4f84-41e5-bc8f-5007cccfcd97","Type":"ContainerStarted","Data":"8445e5270d25a8b56101cf45ee33c9f1cf8b675eac7460316ae6e82bc6847df2"}
Sep 05 02:19:36.594639 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.594207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vtpdm" event={"ID":"553f165a-4f84-41e5-bc8f-5007cccfcd97","Type":"ContainerStarted","Data":"9b58eac73ac29c0192350473361eda8eb0dda3f23612c841757a183c69e9d341"}
Sep 05 02:19:36.608658 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.608629    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-669f5cbcd9-b762l"]
Sep 05 02:19:36.611851 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.611831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.614526 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.614504    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"]
Sep 05 02:19:36.615817 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.615782    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 05 02:19:36.615904 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.615843    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-98jtq\""
Sep 05 02:19:36.617161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.616986    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fonc5a065f9bg\""
Sep 05 02:19:36.617161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.617030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 05 02:19:36.617161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.617051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 05 02:19:36.617161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.616990    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 05 02:19:36.617582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.617564    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.619763 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.619744    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 05 02:19:36.619862 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.619762    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4pgc5\""
Sep 05 02:19:36.623223 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.623115    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vtpdm" podStartSLOduration=2.758424851 podStartE2EDuration="4.623096344s" totalImagesPullingTime="1.769768648s" totalInitContainerRuntime="94.902845ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:36.620849831 +0000 UTC m=+117.042355174" watchObservedRunningTime="2026-09-05 02:19:36.623096344 +0000 UTC m=+117.044601683"
Sep 05 02:19:36.628689 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.628670    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-669f5cbcd9-b762l"]
Sep 05 02:19:36.631079 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.631058    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"]
Sep 05 02:19:36.709732 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.709695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfvdj\" (UniqueName: \"kubernetes.io/projected/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-kube-api-access-hfvdj\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.709880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.709751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-metrics-server-audit-profiles\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.709880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.709861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-secret-metrics-server-tls\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.710016 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.709909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94cd9fea-764b-4d05-8cf8-d9d15b06e25f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-m8f6r\" (UID: \"94cd9fea-764b-4d05-8cf8-d9d15b06e25f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.710086 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.710061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.710139 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.710121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-client-ca-bundle\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.710194 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.710163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-audit-log\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.710231 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.710210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgd5\" (UniqueName: \"kubernetes.io/projected/94cd9fea-764b-4d05-8cf8-d9d15b06e25f-kube-api-access-hqgd5\") pod \"monitoring-plugin-85bbbbbf4-m8f6r\" (UID: \"94cd9fea-764b-4d05-8cf8-d9d15b06e25f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.710267 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.710239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-secret-metrics-server-client-certs\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811584 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfvdj\" (UniqueName: \"kubernetes.io/projected/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-kube-api-access-hfvdj\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-metrics-server-audit-profiles\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-secret-metrics-server-tls\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94cd9fea-764b-4d05-8cf8-d9d15b06e25f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-m8f6r\" (UID: \"94cd9fea-764b-4d05-8cf8-d9d15b06e25f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.811913 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811913 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-client-ca-bundle\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811913 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-audit-log\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.811913 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgd5\" (UniqueName: \"kubernetes.io/projected/94cd9fea-764b-4d05-8cf8-d9d15b06e25f-kube-api-access-hqgd5\") pod \"monitoring-plugin-85bbbbbf4-m8f6r\" (UID: \"94cd9fea-764b-4d05-8cf8-d9d15b06e25f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.811913 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.811862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-secret-metrics-server-client-certs\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.812952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.812416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-audit-log\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.812952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.812730    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-metrics-server-audit-profiles\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.812952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.812910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.814830 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.814803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-secret-metrics-server-client-certs\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.815572 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.815285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94cd9fea-764b-4d05-8cf8-d9d15b06e25f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-m8f6r\" (UID: \"94cd9fea-764b-4d05-8cf8-d9d15b06e25f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.815687 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.815668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-client-ca-bundle\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.815767 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.815741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-secret-metrics-server-tls\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.821171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.821127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfvdj\" (UniqueName: \"kubernetes.io/projected/ee59885b-459e-46cc-9f75-f2a5ae98dfa6-kube-api-access-hfvdj\") pod \"metrics-server-669f5cbcd9-b762l\" (UID: \"ee59885b-459e-46cc-9f75-f2a5ae98dfa6\") " pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.822124 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.822100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgd5\" (UniqueName: \"kubernetes.io/projected/94cd9fea-764b-4d05-8cf8-d9d15b06e25f-kube-api-access-hqgd5\") pod \"monitoring-plugin-85bbbbbf4-m8f6r\" (UID: \"94cd9fea-764b-4d05-8cf8-d9d15b06e25f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:36.925240 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.925159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:36.932377 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:36.932347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:37.414609 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.414479    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"]
Sep 05 02:19:37.418103 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.418070    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-669f5cbcd9-b762l"]
Sep 05 02:19:37.601950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.601906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"}
Sep 05 02:19:37.602365 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.602008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"}
Sep 05 02:19:37.602365 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.602039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"}
Sep 05 02:19:37.644428 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.644382    2582 patch_prober.go:28] interesting pod/image-registry-5c554659c7-k9ljc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:19:37.644614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:37.644453    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" podUID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:19:37.942397 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:37.942366    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee59885b_459e_46cc_9f75_f2a5ae98dfa6.slice/crio-1b027d0a7c786349fc92a0774b87f2ad005660ed97fe24b0982e0e934d4c27fb WatchSource:0}: Error finding container 1b027d0a7c786349fc92a0774b87f2ad005660ed97fe24b0982e0e934d4c27fb: Status 404 returned error can't find the container with id 1b027d0a7c786349fc92a0774b87f2ad005660ed97fe24b0982e0e934d4c27fb
Sep 05 02:19:38.609717 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.609671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l" event={"ID":"ee59885b-459e-46cc-9f75-f2a5ae98dfa6","Type":"ContainerStarted","Data":"1b027d0a7c786349fc92a0774b87f2ad005660ed97fe24b0982e0e934d4c27fb"}
Sep 05 02:19:38.611273 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.611237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r" event={"ID":"94cd9fea-764b-4d05-8cf8-d9d15b06e25f","Type":"ContainerStarted","Data":"2f0c1a88ecc4a222665b4c6f1350740fe32b9298c647333dc2c95734a1827b3f"}
Sep 05 02:19:38.615608 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.615555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"}
Sep 05 02:19:38.615734 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.615669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"}
Sep 05 02:19:38.618466 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.618442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"0df03239a54b66a0104b2c160888239e66531b07ced4d9728ea1b661d7c48f72"}
Sep 05 02:19:38.618597 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.618507    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"2588fc22a7a0a757e1722b1b235b48785b37040cabcd50d118041e8efc25be01"}
Sep 05 02:19:38.618597 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:38.618537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"e43b8180385fa82b2c1fd854e486b0558af6eb7c821bc764d54ed203c0c17cde"}
Sep 05 02:19:39.489893 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:39.489822    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:19:39.626596 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:39.626556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"3b2414c58d933fd087c57df526d179b1bf66ded1c4b0bdcb698f07da44bac8d7"}
Sep 05 02:19:39.627055 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:39.626631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"d3ec6123a267f90ef18d43f8d24cd202b4ef539f124cd6c51b84c17d94bfc0cb"}
Sep 05 02:19:39.630647 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:39.630614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerStarted","Data":"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"}
Sep 05 02:19:39.673455 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:39.673398    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.8645211760000002 podStartE2EDuration="7.673380216s" totalImagesPullingTime="3.790904835s" totalInitContainerRuntime="17.954205ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:39.67286697 +0000 UTC m=+120.094372307" watchObservedRunningTime="2026-09-05 02:19:39.673380216 +0000 UTC m=+120.094885594"
Sep 05 02:19:40.635935 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.635897    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r" event={"ID":"94cd9fea-764b-4d05-8cf8-d9d15b06e25f","Type":"ContainerStarted","Data":"65cfe406bc5ff09c92f3f044ff9ef47121ea2579ff3169b5e5b613f3c5e7253f"}
Sep 05 02:19:40.636396 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.635961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:40.639333 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.639304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" event={"ID":"a8f9d653-c15b-4d51-bbde-2b05b30fef65","Type":"ContainerStarted","Data":"136de29e5678494b5d528ddfa1ef69a5336e2f7b37cb8b49807341265af210b4"}
Sep 05 02:19:40.639477 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.639410    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:40.641107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.641057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l" event={"ID":"ee59885b-459e-46cc-9f75-f2a5ae98dfa6","Type":"ContainerStarted","Data":"43708b77113962eb04511802af0507b13788ce977da2aaec89d0059559c8fb3f"}
Sep 05 02:19:40.642095 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.642076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r"
Sep 05 02:19:40.657191 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.657150    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-m8f6r" podStartSLOduration=2.710682886 podStartE2EDuration="4.657137483s" totalImagesPullingTime="1.946454597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:40.656918428 +0000 UTC m=+121.078423768" watchObservedRunningTime="2026-09-05 02:19:40.657137483 +0000 UTC m=+121.078642822"
Sep 05 02:19:40.684938 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.684891    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2" podStartSLOduration=3.451785411 podStartE2EDuration="6.684877771s" totalImagesPullingTime="3.23309236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:40.684403599 +0000 UTC m=+121.105908941" watchObservedRunningTime="2026-09-05 02:19:40.684877771 +0000 UTC m=+121.106383109"
Sep 05 02:19:40.707769 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:40.707726    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l" podStartSLOduration=2.762413486 podStartE2EDuration="4.707711426s" totalImagesPullingTime="1.94529794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:40.706535692 +0000 UTC m=+121.128041030" watchObservedRunningTime="2026-09-05 02:19:40.707711426 +0000 UTC m=+121.129216764"
Sep 05 02:19:44.600450 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:44.600414    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c554659c7-k9ljc"]
Sep 05 02:19:46.654164 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:46.654137    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-db4fbb5df-gwzz2"
Sep 05 02:19:49.829895 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:49.829849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:19:49.832197 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:49.832166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f623e00-8e5e-4eb1-8f53-3751e8474c52-metrics-certs\") pod \"network-metrics-daemon-qkm56\" (UID: \"9f623e00-8e5e-4eb1-8f53-3751e8474c52\") " pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:19:49.925710 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:49.925674    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4nzpg\""
Sep 05 02:19:49.932943 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:49.932922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qkm56"
Sep 05 02:19:50.214689 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:50.214617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qkm56"]
Sep 05 02:19:50.217200 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:19:50.217173    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f623e00_8e5e_4eb1_8f53_3751e8474c52.slice/crio-9e9f3faa181b934cec372147bece00b3b5c66bbe960f2060e0a2b554bbab5b50 WatchSource:0}: Error finding container 9e9f3faa181b934cec372147bece00b3b5c66bbe960f2060e0a2b554bbab5b50: Status 404 returned error can't find the container with id 9e9f3faa181b934cec372147bece00b3b5c66bbe960f2060e0a2b554bbab5b50
Sep 05 02:19:50.681484 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:50.681444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkm56" event={"ID":"9f623e00-8e5e-4eb1-8f53-3751e8474c52","Type":"ContainerStarted","Data":"9e9f3faa181b934cec372147bece00b3b5c66bbe960f2060e0a2b554bbab5b50"}
Sep 05 02:19:51.687186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:51.687145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkm56" event={"ID":"9f623e00-8e5e-4eb1-8f53-3751e8474c52","Type":"ContainerStarted","Data":"f254151ad2026c59c3cf658671f2d34beb5ff412673a4c9f51e4658642ee8a05"}
Sep 05 02:19:51.687602 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:51.687225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qkm56" event={"ID":"9f623e00-8e5e-4eb1-8f53-3751e8474c52","Type":"ContainerStarted","Data":"38f60904ee611a46acf1623f5a383dc2de4c5e705252d2d054b0828ac03fb85e"}
Sep 05 02:19:56.925433 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:56.925395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:19:56.925892 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:19:56.925492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:20:07.661815 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.661757    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qkm56" podStartSLOduration=146.606507654 podStartE2EDuration="2m27.661740675s" totalImagesPullingTime="1.055233021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:51.709150569 +0000 UTC m=+132.130655908" watchObservedRunningTime="2026-09-05 02:20:07.661740675 +0000 UTC m=+148.083246014"
Sep 05 02:20:07.662472 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.662454    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f58f69bb-7hjhv"]
Sep 05 02:20:07.681226 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.681193    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f58f69bb-7hjhv"]
Sep 05 02:20:07.681400 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.681294    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.685710 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.685656    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:20:07.687143 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.687117    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:20:07.687290 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.687179    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:20:07.687868 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.687852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4hkzj\""
Sep 05 02:20:07.688430 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.688410    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:20:07.688545 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.688439    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:20:07.688545 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.688521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:20:07.689067 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.688859    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:20:07.691562 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.691545    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 05 02:20:07.771208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-oauth-config\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.771208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-serving-cert\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.771461 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sk4l\" (UniqueName: \"kubernetes.io/projected/bf69fbf0-3136-42e7-8632-08a20c55545a-kube-api-access-5sk4l\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.771461 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-console-config\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.771461 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-service-ca\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.771461 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-oauth-serving-cert\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.771461 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.771457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-trusted-ca-bundle\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872524 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-oauth-config\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872524 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-serving-cert\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872782 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sk4l\" (UniqueName: \"kubernetes.io/projected/bf69fbf0-3136-42e7-8632-08a20c55545a-kube-api-access-5sk4l\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872782 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-console-config\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872782 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-service-ca\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872782 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-oauth-serving-cert\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.872782 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.872653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-trusted-ca-bundle\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.873475 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.873445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-oauth-serving-cert\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.873597 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.873499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-console-config\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.873597 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.873505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-service-ca\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.873836 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.873816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-trusted-ca-bundle\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.875034 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.875008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-oauth-config\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.875131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.875114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-serving-cert\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.881807 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.881789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sk4l\" (UniqueName: \"kubernetes.io/projected/bf69fbf0-3136-42e7-8632-08a20c55545a-kube-api-access-5sk4l\") pod \"console-7f58f69bb-7hjhv\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") " pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:07.992581 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:07.992479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:08.308341 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:08.308316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f58f69bb-7hjhv"]
Sep 05 02:20:08.310911 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:20:08.310883    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf69fbf0_3136_42e7_8632_08a20c55545a.slice/crio-eb1f312436519ec945bf50519423445ac616e02e2e084398fa89942d4e202df2 WatchSource:0}: Error finding container eb1f312436519ec945bf50519423445ac616e02e2e084398fa89942d4e202df2: Status 404 returned error can't find the container with id eb1f312436519ec945bf50519423445ac616e02e2e084398fa89942d4e202df2
Sep 05 02:20:08.761685 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:08.761647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f58f69bb-7hjhv" event={"ID":"bf69fbf0-3136-42e7-8632-08a20c55545a","Type":"ContainerStarted","Data":"eb1f312436519ec945bf50519423445ac616e02e2e084398fa89942d4e202df2"}
Sep 05 02:20:09.620241 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.620181    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" podUID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" containerName="registry" containerID="cri-o://345356cb43b02ead9dcfc7836771aca5c60aaeda090014fb3874a435b09c98a5" gracePeriod=30
Sep 05 02:20:09.767775 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.767742    2582 generic.go:356] "Generic (PLEG): container finished" podID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" containerID="345356cb43b02ead9dcfc7836771aca5c60aaeda090014fb3874a435b09c98a5" exitCode=0
Sep 05 02:20:09.768142 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.767813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" event={"ID":"38e3cd40-d180-4e9d-9ac9-9b004615c1bf","Type":"ContainerDied","Data":"345356cb43b02ead9dcfc7836771aca5c60aaeda090014fb3874a435b09c98a5"}
Sep 05 02:20:09.863046 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.863024    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:20:09.992320 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992221    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49zjm\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-kube-api-access-49zjm\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992320 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-bound-sa-token\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992320 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992305    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-image-registry-private-configuration\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992320 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992653 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992354    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-certificates\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992653 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992388    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-installation-pull-secrets\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992653 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-ca-trust-extracted\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992653 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992544    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-trusted-ca\") pod \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\" (UID: \"38e3cd40-d180-4e9d-9ac9-9b004615c1bf\") "
Sep 05 02:20:09.992891 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.992860    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:09.993197 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.993169    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:09.994889 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.994853    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:09.995028 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.994889    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:09.995028 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.994983    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-kube-api-access-49zjm" (OuterVolumeSpecName: "kube-api-access-49zjm") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "kube-api-access-49zjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:09.995028 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.995013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:09.995211 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:09.995108    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:10.001454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.001432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "38e3cd40-d180-4e9d-9ac9-9b004615c1bf" (UID: "38e3cd40-d180-4e9d-9ac9-9b004615c1bf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:20:10.093245 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093218    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-certificates\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093245 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093244    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-installation-pull-secrets\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093259    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-ca-trust-extracted\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093269    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-trusted-ca\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093278    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zjm\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-kube-api-access-49zjm\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093286    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-bound-sa-token\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093295    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-image-registry-private-configuration\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.093452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.093303    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38e3cd40-d180-4e9d-9ac9-9b004615c1bf-registry-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:10.773286 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.773247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc" event={"ID":"38e3cd40-d180-4e9d-9ac9-9b004615c1bf","Type":"ContainerDied","Data":"a6b0ea2120068dca0d23add09d32b8b9b4a32118cc17760022c0aae4ade70f60"}
Sep 05 02:20:10.773730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.773348    2582 scope.go:117] "RemoveContainer" containerID="345356cb43b02ead9dcfc7836771aca5c60aaeda090014fb3874a435b09c98a5"
Sep 05 02:20:10.773730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.773267    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c554659c7-k9ljc"
Sep 05 02:20:10.794629 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.794602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c554659c7-k9ljc"]
Sep 05 02:20:10.799901 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:10.799871    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c554659c7-k9ljc"]
Sep 05 02:20:12.109506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:12.109475    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" path="/var/lib/kubelet/pods/38e3cd40-d180-4e9d-9ac9-9b004615c1bf/volumes"
Sep 05 02:20:14.791819 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:14.791774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f58f69bb-7hjhv" event={"ID":"bf69fbf0-3136-42e7-8632-08a20c55545a","Type":"ContainerStarted","Data":"e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f"}
Sep 05 02:20:14.813872 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:14.813818    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f58f69bb-7hjhv" podStartSLOduration=1.851221904 podStartE2EDuration="7.813800879s" totalImagesPullingTime="5.962578975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:14.811572937 +0000 UTC m=+155.233078275" watchObservedRunningTime="2026-09-05 02:20:14.813800879 +0000 UTC m=+155.235306217"
Sep 05 02:20:16.930807 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:16.930777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:20:16.934444 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:16.934424    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-669f5cbcd9-b762l"
Sep 05 02:20:17.993192 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:17.993159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:17.993560 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:17.993220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:17.997813 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:17.997792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:18.810901 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:18.810874    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:20:52.241637 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.241596    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:52.242161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.241882    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="alertmanager" containerID="cri-o://80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa" gracePeriod=120
Sep 05 02:20:52.242161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.241958    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-metric" containerID="cri-o://27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d" gracePeriod=120
Sep 05 02:20:52.242161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.241958    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-web" containerID="cri-o://e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a" gracePeriod=120
Sep 05 02:20:52.242161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.242023    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy" containerID="cri-o://33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949" gracePeriod=120
Sep 05 02:20:52.242161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.241974    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="config-reloader" containerID="cri-o://5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27" gracePeriod=120
Sep 05 02:20:52.242161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.242030    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="prom-label-proxy" containerID="cri-o://35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532" gracePeriod=120
Sep 05 02:20:52.939980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.939939    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532" exitCode=0
Sep 05 02:20:52.939980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.939962    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949" exitCode=0
Sep 05 02:20:52.939980 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.939983    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27" exitCode=0
Sep 05 02:20:52.940205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.939992    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa" exitCode=0
Sep 05 02:20:52.940205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.939996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"}
Sep 05 02:20:52.940205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.940057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"}
Sep 05 02:20:52.940205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.940082    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"}
Sep 05 02:20:52.940205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:52.940107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"}
Sep 05 02:20:53.501508 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.501483    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:53.577695 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577659    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-cluster-tls-config\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.577880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577727    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-config-out\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.577880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577758    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-main-tls\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.577880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577793    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-web-config\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.577880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-main-db\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.577880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-metric\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.578241 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.578209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:20:53.578554 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.577921    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-web\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.578821 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.578805    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-metrics-client-ca\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.579458 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.579433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd5sj\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-kube-api-access-wd5sj\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.580078 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.579640    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-trusted-ca-bundle\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.580239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.580219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.580342 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.580285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-config-volume\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.580342 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.580332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-tls-assets\") pod \"75e10383-4fba-4499-8a0b-ce741fe41e71\" (UID: \"75e10383-4fba-4499-8a0b-ce741fe41e71\") "
Sep 05 02:20:53.580737 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.580577    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-main-db\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.580737 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.579267    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:53.580737 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.580051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:53.580737 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.580711    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.582306 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.582266    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-config-out" (OuterVolumeSpecName: "config-out") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:20:53.582398 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.582315    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.582745 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.582718    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-kube-api-access-wd5sj" (OuterVolumeSpecName: "kube-api-access-wd5sj") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "kube-api-access-wd5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:53.582996 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.582959    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-config-volume" (OuterVolumeSpecName: "config-volume") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.583308 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.583291    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:53.583386 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.583326    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.583898 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.583870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.586282 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.586177    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.592130 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.592107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-web-config" (OuterVolumeSpecName: "web-config") pod "75e10383-4fba-4499-8a0b-ce741fe41e71" (UID: "75e10383-4fba-4499-8a0b-ce741fe41e71"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:53.681109 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681075    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-metrics-client-ca\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681109 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681103    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd5sj\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-kube-api-access-wd5sj\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681109 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681112    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e10383-4fba-4499-8a0b-ce741fe41e71-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681124    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681132    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-config-volume\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681141    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75e10383-4fba-4499-8a0b-ce741fe41e71-tls-assets\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681150    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-cluster-tls-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681158    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75e10383-4fba-4499-8a0b-ce741fe41e71-config-out\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681166    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-main-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681174    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-web-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681183    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.681332 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.681191    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75e10383-4fba-4499-8a0b-ce741fe41e71-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:20:53.948246 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948156    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d" exitCode=0
Sep 05 02:20:53.948246 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948185    2582 generic.go:356] "Generic (PLEG): container finished" podID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerID="e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a" exitCode=0
Sep 05 02:20:53.948246 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948231    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:53.948442 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"}
Sep 05 02:20:53.948442 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"}
Sep 05 02:20:53.948442 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75e10383-4fba-4499-8a0b-ce741fe41e71","Type":"ContainerDied","Data":"236ec73ce8a9456c686ae0c0829d5317cf45bd735aa1f99278f97e713adbdf4d"}
Sep 05 02:20:53.948442 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.948366    2582 scope.go:117] "RemoveContainer" containerID="35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"
Sep 05 02:20:53.956265 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.956245    2582 scope.go:117] "RemoveContainer" containerID="27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"
Sep 05 02:20:53.964684 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.964663    2582 scope.go:117] "RemoveContainer" containerID="33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"
Sep 05 02:20:53.973536 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.973517    2582 scope.go:117] "RemoveContainer" containerID="e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"
Sep 05 02:20:53.973536 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.973527    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:53.979454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.979430    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:53.981455 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.981436    2582 scope.go:117] "RemoveContainer" containerID="5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"
Sep 05 02:20:53.989453 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.989435    2582 scope.go:117] "RemoveContainer" containerID="80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"
Sep 05 02:20:53.996503 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:53.996486    2582 scope.go:117] "RemoveContainer" containerID="bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"
Sep 05 02:20:54.003470 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.003451    2582 scope.go:117] "RemoveContainer" containerID="35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"
Sep 05 02:20:54.003756 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.003737    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532\": container with ID starting with 35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532 not found: ID does not exist" containerID="35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"
Sep 05 02:20:54.003798 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.003765    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"} err="failed to get container status \"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532\": rpc error: code = NotFound desc = could not find container \"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532\": container with ID starting with 35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532 not found: ID does not exist"
Sep 05 02:20:54.003798 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.003796    2582 scope.go:117] "RemoveContainer" containerID="27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"
Sep 05 02:20:54.004078 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.004030    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d\": container with ID starting with 27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d not found: ID does not exist" containerID="27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"
Sep 05 02:20:54.004166 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004086    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"} err="failed to get container status \"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d\": rpc error: code = NotFound desc = could not find container \"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d\": container with ID starting with 27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d not found: ID does not exist"
Sep 05 02:20:54.004166 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004108    2582 scope.go:117] "RemoveContainer" containerID="33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"
Sep 05 02:20:54.004331 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.004308    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949\": container with ID starting with 33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949 not found: ID does not exist" containerID="33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"
Sep 05 02:20:54.004368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004335    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"} err="failed to get container status \"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949\": rpc error: code = NotFound desc = could not find container \"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949\": container with ID starting with 33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949 not found: ID does not exist"
Sep 05 02:20:54.004368 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004347    2582 scope.go:117] "RemoveContainer" containerID="e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"
Sep 05 02:20:54.004553 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.004536    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a\": container with ID starting with e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a not found: ID does not exist" containerID="e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"
Sep 05 02:20:54.004614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004560    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"} err="failed to get container status \"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a\": rpc error: code = NotFound desc = could not find container \"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a\": container with ID starting with e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a not found: ID does not exist"
Sep 05 02:20:54.004614 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004579    2582 scope.go:117] "RemoveContainer" containerID="5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"
Sep 05 02:20:54.004820 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.004805    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27\": container with ID starting with 5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27 not found: ID does not exist" containerID="5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"
Sep 05 02:20:54.004861 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004823    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"} err="failed to get container status \"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27\": rpc error: code = NotFound desc = could not find container \"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27\": container with ID starting with 5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27 not found: ID does not exist"
Sep 05 02:20:54.004861 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.004837    2582 scope.go:117] "RemoveContainer" containerID="80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"
Sep 05 02:20:54.005091 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.005073    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa\": container with ID starting with 80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa not found: ID does not exist" containerID="80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"
Sep 05 02:20:54.005161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005096    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"} err="failed to get container status \"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa\": rpc error: code = NotFound desc = could not find container \"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa\": container with ID starting with 80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa not found: ID does not exist"
Sep 05 02:20:54.005161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005114    2582 scope.go:117] "RemoveContainer" containerID="bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"
Sep 05 02:20:54.005323 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:20:54.005307    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0\": container with ID starting with bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0 not found: ID does not exist" containerID="bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"
Sep 05 02:20:54.005364 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005325    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"} err="failed to get container status \"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0\": rpc error: code = NotFound desc = could not find container \"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0\": container with ID starting with bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0 not found: ID does not exist"
Sep 05 02:20:54.005364 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005336    2582 scope.go:117] "RemoveContainer" containerID="35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"
Sep 05 02:20:54.005549 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005532    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532"} err="failed to get container status \"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532\": rpc error: code = NotFound desc = could not find container \"35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532\": container with ID starting with 35ac651cb19c298d71401ff92391b509e6e2465676f6ed1ca321c7ed465d9532 not found: ID does not exist"
Sep 05 02:20:54.005621 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005550    2582 scope.go:117] "RemoveContainer" containerID="27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"
Sep 05 02:20:54.005769 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005754    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d"} err="failed to get container status \"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d\": rpc error: code = NotFound desc = could not find container \"27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d\": container with ID starting with 27942ce47a164606488b329db7e9955418f2d95ee6db4ea2baa1997c050ee47d not found: ID does not exist"
Sep 05 02:20:54.005814 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005769    2582 scope.go:117] "RemoveContainer" containerID="33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"
Sep 05 02:20:54.005957 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005940    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949"} err="failed to get container status \"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949\": rpc error: code = NotFound desc = could not find container \"33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949\": container with ID starting with 33f86fcd327f29557412bf20accf886862f2823f16838064b4455c35794a5949 not found: ID does not exist"
Sep 05 02:20:54.006045 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.005958    2582 scope.go:117] "RemoveContainer" containerID="e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"
Sep 05 02:20:54.006209 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006192    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a"} err="failed to get container status \"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a\": rpc error: code = NotFound desc = could not find container \"e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a\": container with ID starting with e2a78a5413ec2902e7bd745d479368242601ce9a62d376ad56476605fa7b897a not found: ID does not exist"
Sep 05 02:20:54.006277 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006211    2582 scope.go:117] "RemoveContainer" containerID="5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"
Sep 05 02:20:54.006471 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006452    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27"} err="failed to get container status \"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27\": rpc error: code = NotFound desc = could not find container \"5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27\": container with ID starting with 5c85df2e204abe69afc796b263129130cdaa3133b4972c538f5f52a31f733b27 not found: ID does not exist"
Sep 05 02:20:54.006517 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006474    2582 scope.go:117] "RemoveContainer" containerID="80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"
Sep 05 02:20:54.006701 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006685    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa"} err="failed to get container status \"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa\": rpc error: code = NotFound desc = could not find container \"80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa\": container with ID starting with 80d901b1ea45733c6e8916b4271b00b066dc6541e10ff9b1200b641d18c567aa not found: ID does not exist"
Sep 05 02:20:54.006769 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006703    2582 scope.go:117] "RemoveContainer" containerID="bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"
Sep 05 02:20:54.006905 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.006890    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0"} err="failed to get container status \"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0\": rpc error: code = NotFound desc = could not find container \"bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0\": container with ID starting with bac0c0eb7ca04d3e8fd4b2d3946e3971c9a78aa33dba358b998001fa439244c0 not found: ID does not exist"
Sep 05 02:20:54.009285 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009266    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:54.009537 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009526    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="alertmanager"
Sep 05 02:20:54.009578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009539    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="alertmanager"
Sep 05 02:20:54.009578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009549    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" containerName="registry"
Sep 05 02:20:54.009578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009556    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" containerName="registry"
Sep 05 02:20:54.009578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-metric"
Sep 05 02:20:54.009578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009570    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-metric"
Sep 05 02:20:54.009578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009577    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="prom-label-proxy"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009594    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="prom-label-proxy"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-web"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009607    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-web"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009613    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="init-config-reloader"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009619    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="init-config-reloader"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009624    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="config-reloader"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009629    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="config-reloader"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009639    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009644    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009689    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e3cd40-d180-4e9d-9ac9-9b004615c1bf" containerName="registry"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009695    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="prom-label-proxy"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009702    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="config-reloader"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009709    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009714    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-metric"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009720    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="alertmanager"
Sep 05 02:20:54.009770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.009724    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" containerName="kube-rbac-proxy-web"
Sep 05 02:20:54.013835 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.013819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.016619 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.016592    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.017420    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-szr26\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.017447    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.017604    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.019173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.019758    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.020040    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 05 02:20:54.020224 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.020137    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 05 02:20:54.020644 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.020567    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 05 02:20:54.025569 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.025548    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 05 02:20:54.027859 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.027840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:54.084733 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b324fcee-86cf-4848-a906-8fc42871cd5d-config-out\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.084899 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.084899 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b324fcee-86cf-4848-a906-8fc42871cd5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.084899 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.084899 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-web-config\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.084899 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b324fcee-86cf-4848-a906-8fc42871cd5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085094 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzpq\" (UniqueName: \"kubernetes.io/projected/b324fcee-86cf-4848-a906-8fc42871cd5d-kube-api-access-krzpq\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085094 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.084953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b324fcee-86cf-4848-a906-8fc42871cd5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085094 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.085010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085094 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.085063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b324fcee-86cf-4848-a906-8fc42871cd5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085094 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.085086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085238 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.085103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.085238 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.085162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.108564 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.108534    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e10383-4fba-4499-8a0b-ce741fe41e71" path="/var/lib/kubelet/pods/75e10383-4fba-4499-8a0b-ce741fe41e71/volumes"
Sep 05 02:20:54.185844 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.185805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.185844 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.185840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b324fcee-86cf-4848-a906-8fc42871cd5d-config-out\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.185880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.185904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b324fcee-86cf-4848-a906-8fc42871cd5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.185933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.185993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-web-config\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b324fcee-86cf-4848-a906-8fc42871cd5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzpq\" (UniqueName: \"kubernetes.io/projected/b324fcee-86cf-4848-a906-8fc42871cd5d-kube-api-access-krzpq\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b324fcee-86cf-4848-a906-8fc42871cd5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186451 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b324fcee-86cf-4848-a906-8fc42871cd5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186451 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.186451 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.186183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.187253 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.187224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b324fcee-86cf-4848-a906-8fc42871cd5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.187514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.187489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b324fcee-86cf-4848-a906-8fc42871cd5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.188789 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.188708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b324fcee-86cf-4848-a906-8fc42871cd5d-config-out\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189344 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.188991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189344 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.189023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189344 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.189080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189344 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.189295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189344 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.189303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189563 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.189446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b324fcee-86cf-4848-a906-8fc42871cd5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.189563 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.189498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b324fcee-86cf-4848-a906-8fc42871cd5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.190234 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.190213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.190835 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.190819    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b324fcee-86cf-4848-a906-8fc42871cd5d-web-config\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.198499 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.198459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzpq\" (UniqueName: \"kubernetes.io/projected/b324fcee-86cf-4848-a906-8fc42871cd5d-kube-api-access-krzpq\") pod \"alertmanager-main-0\" (UID: \"b324fcee-86cf-4848-a906-8fc42871cd5d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.328040 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.328007    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 05 02:20:54.607077 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.607051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 05 02:20:54.609027 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:20:54.609001    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb324fcee_86cf_4848_a906_8fc42871cd5d.slice/crio-5b4814570d5123287cb9367e4f1b4fdbd33de2956fe2ef8b99ef381f7f823551 WatchSource:0}: Error finding container 5b4814570d5123287cb9367e4f1b4fdbd33de2956fe2ef8b99ef381f7f823551: Status 404 returned error can't find the container with id 5b4814570d5123287cb9367e4f1b4fdbd33de2956fe2ef8b99ef381f7f823551
Sep 05 02:20:54.954100 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.954019    2582 generic.go:356] "Generic (PLEG): container finished" podID="b324fcee-86cf-4848-a906-8fc42871cd5d" containerID="9b9d6e27f57bc65388060101d6b2e3d19a29808ee46e7007b7de596480ef42d6" exitCode=0
Sep 05 02:20:54.954100 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.954063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerDied","Data":"9b9d6e27f57bc65388060101d6b2e3d19a29808ee46e7007b7de596480ef42d6"}
Sep 05 02:20:54.954280 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:54.954114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"5b4814570d5123287cb9367e4f1b4fdbd33de2956fe2ef8b99ef381f7f823551"}
Sep 05 02:20:55.961553 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.961510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"84ee549f99cf37dbccfea1847f45c96e7c94e471d02cb6ac9b410688c2abe4c6"}
Sep 05 02:20:55.961925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.961611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"0b0b3dbeafb509b13f095bfec7d1f9f767fb3e5705799738d12fa623800c05c1"}
Sep 05 02:20:55.961925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.961642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"539d38881baa05a0f365d103c1d0032224ab2e5b5fe7f8ba9d17812e8da4f606"}
Sep 05 02:20:55.961925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.961680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"91c3509919b534811ff076110ceea8518d69d34f19374d703d670fddc99bc9ba"}
Sep 05 02:20:55.961925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.961704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"7443c14d82c00f3e456e6b23345e24426aa468f7e7de90c66b3fffbfa785b90b"}
Sep 05 02:20:55.961925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.961729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b324fcee-86cf-4848-a906-8fc42871cd5d","Type":"ContainerStarted","Data":"2d94e047afb88e7d72dc3046bc5fe4c94dec2a1b5568badfc8c42c0cd7fe3ea4"}
Sep 05 02:20:55.988635 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:55.988581    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.969961891 podStartE2EDuration="2.988566685s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.604794ms" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:55.987582123 +0000 UTC m=+196.409087463" watchObservedRunningTime="2026-09-05 02:20:55.988566685 +0000 UTC m=+196.410072024"
Sep 05 02:20:56.290345 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.290265    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d78687b65-dc25w"]
Sep 05 02:20:56.293916 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.293894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.296760 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.296730    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 05 02:20:56.296875 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.296790    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 05 02:20:56.296875 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.296852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 05 02:20:56.296982 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.296947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 05 02:20:56.297083 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.297067    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 05 02:20:56.297164 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.297150    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kmp7k\""
Sep 05 02:20:56.301382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.301362    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 05 02:20:56.307029 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.307004    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d78687b65-dc25w"]
Sep 05 02:20:56.406781 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.406741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407001 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.406789    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-serving-certs-ca-bundle\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407001 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.406811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-telemeter-client-tls\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407001 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.406909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-secret-telemeter-client\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.407017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-federate-client-tls\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.407050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-metrics-client-ca\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.407067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.407222 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.407126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2rn\" (UniqueName: \"kubernetes.io/projected/fa3e561e-aef3-4000-86cc-69d56b83e625-kube-api-access-kn2rn\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508280 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-serving-certs-ca-bundle\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-telemeter-client-tls\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-secret-telemeter-client\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508619 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-federate-client-tls\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508619 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-metrics-client-ca\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508619 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.508619 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.508563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2rn\" (UniqueName: \"kubernetes.io/projected/fa3e561e-aef3-4000-86cc-69d56b83e625-kube-api-access-kn2rn\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.509258 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.509228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-serving-certs-ca-bundle\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.509477 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.509450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.509831 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.509808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa3e561e-aef3-4000-86cc-69d56b83e625-metrics-client-ca\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.510900 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.510875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.511063 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.511043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-federate-client-tls\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.511175 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.511156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-telemeter-client-tls\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.511211 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.511177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fa3e561e-aef3-4000-86cc-69d56b83e625-secret-telemeter-client\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.516348 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.516329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2rn\" (UniqueName: \"kubernetes.io/projected/fa3e561e-aef3-4000-86cc-69d56b83e625-kube-api-access-kn2rn\") pod \"telemeter-client-5d78687b65-dc25w\" (UID: \"fa3e561e-aef3-4000-86cc-69d56b83e625\") " pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.604169 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.604134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w"
Sep 05 02:20:56.884125 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.884004    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d78687b65-dc25w"]
Sep 05 02:20:56.886864 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:20:56.886832    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3e561e_aef3_4000_86cc_69d56b83e625.slice/crio-d6ec9f47c6df742f46d75b42936b2c4c29651ca5ddd5e0a10507a6d680414770 WatchSource:0}: Error finding container d6ec9f47c6df742f46d75b42936b2c4c29651ca5ddd5e0a10507a6d680414770: Status 404 returned error can't find the container with id d6ec9f47c6df742f46d75b42936b2c4c29651ca5ddd5e0a10507a6d680414770
Sep 05 02:20:56.966763 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:56.966728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w" event={"ID":"fa3e561e-aef3-4000-86cc-69d56b83e625","Type":"ContainerStarted","Data":"d6ec9f47c6df742f46d75b42936b2c4c29651ca5ddd5e0a10507a6d680414770"}
Sep 05 02:20:58.977453 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:58.977413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w" event={"ID":"fa3e561e-aef3-4000-86cc-69d56b83e625","Type":"ContainerStarted","Data":"208f534eee11ab6bd0409b7c676e35138bc1b63400d88061cd2ef90f192027f3"}
Sep 05 02:20:58.977878 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:58.977506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w" event={"ID":"fa3e561e-aef3-4000-86cc-69d56b83e625","Type":"ContainerStarted","Data":"45eb2089acac795536761fa2dd89ea592f2451fae98378c4099a6c22c2033ee0"}
Sep 05 02:20:58.977878 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:58.977530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w" event={"ID":"fa3e561e-aef3-4000-86cc-69d56b83e625","Type":"ContainerStarted","Data":"167b50b1d7082db5186f3a6e948855f9127f0d0a1221d086d7d0fff17726f465"}
Sep 05 02:20:59.002280 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.002179    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d78687b65-dc25w" podStartSLOduration=1.157167397 podStartE2EDuration="3.00216318s" totalImagesPullingTime="1.844995783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:59.000009751 +0000 UTC m=+199.421515116" watchObservedRunningTime="2026-09-05 02:20:59.00216318 +0000 UTC m=+199.423668519"
Sep 05 02:20:59.095296 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.095262    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d54df6f9c-pvs7c"]
Sep 05 02:20:59.098717 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.098699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.111234 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.111207    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d54df6f9c-pvs7c"]
Sep 05 02:20:59.236303 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-trusted-ca-bundle\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.236488 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7jb\" (UniqueName: \"kubernetes.io/projected/01f07e51-c6cc-4221-a1cb-b3b866e0e842-kube-api-access-fz7jb\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.236488 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-config\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.236577 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-oauth-config\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.236577 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-oauth-serving-cert\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.236577 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-serving-cert\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.236577 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.236570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-service-ca\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337053 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-oauth-config\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-oauth-serving-cert\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-serving-cert\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-service-ca\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-trusted-ca-bundle\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7jb\" (UniqueName: \"kubernetes.io/projected/01f07e51-c6cc-4221-a1cb-b3b866e0e842-kube-api-access-fz7jb\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337424 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-config\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.337891 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-oauth-serving-cert\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.338049 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.337919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-config\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.338049 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.338005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-trusted-ca-bundle\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.338239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.338056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-service-ca\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.339541 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.339511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-serving-cert\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.339689 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.339665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-oauth-config\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.345068 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.345047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7jb\" (UniqueName: \"kubernetes.io/projected/01f07e51-c6cc-4221-a1cb-b3b866e0e842-kube-api-access-fz7jb\") pod \"console-d54df6f9c-pvs7c\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") " pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.407171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.407116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:20:59.679422 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.679397    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d54df6f9c-pvs7c"]
Sep 05 02:20:59.681812 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:20:59.681784    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f07e51_c6cc_4221_a1cb_b3b866e0e842.slice/crio-4083a5a2198aed2dc9b284a122eb8686f306e32e4ca5fd732d33cc1a129a577a WatchSource:0}: Error finding container 4083a5a2198aed2dc9b284a122eb8686f306e32e4ca5fd732d33cc1a129a577a: Status 404 returned error can't find the container with id 4083a5a2198aed2dc9b284a122eb8686f306e32e4ca5fd732d33cc1a129a577a
Sep 05 02:20:59.984636 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.984541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d54df6f9c-pvs7c" event={"ID":"01f07e51-c6cc-4221-a1cb-b3b866e0e842","Type":"ContainerStarted","Data":"e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7"}
Sep 05 02:20:59.984636 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:20:59.984594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d54df6f9c-pvs7c" event={"ID":"01f07e51-c6cc-4221-a1cb-b3b866e0e842","Type":"ContainerStarted","Data":"4083a5a2198aed2dc9b284a122eb8686f306e32e4ca5fd732d33cc1a129a577a"}
Sep 05 02:21:00.003834 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:00.003781    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d54df6f9c-pvs7c" podStartSLOduration=1.003765918 podStartE2EDuration="1.003765918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:00.003116743 +0000 UTC m=+200.424622110" watchObservedRunningTime="2026-09-05 02:21:00.003765918 +0000 UTC m=+200.425271260"
Sep 05 02:21:09.408345 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:09.408306    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:21:09.408841 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:09.408364    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:21:09.413077 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:09.413055    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:21:10.029404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:10.029374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:21:10.078030 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:10.077994    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f58f69bb-7hjhv"]
Sep 05 02:21:35.102161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.102099    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f58f69bb-7hjhv" podUID="bf69fbf0-3136-42e7-8632-08a20c55545a" containerName="console" containerID="cri-o://e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f" gracePeriod=15
Sep 05 02:21:35.345216 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.345192    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:21:35.447829 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.447749    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-oauth-config\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.447829 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.447799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-oauth-serving-cert\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.448052 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.447837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-serving-cert\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.448052 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-trusted-ca-bundle\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.448149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448055    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sk4l\" (UniqueName: \"kubernetes.io/projected/bf69fbf0-3136-42e7-8632-08a20c55545a-kube-api-access-5sk4l\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.448149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-console-config\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.448149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448118    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-service-ca\") pod \"bf69fbf0-3136-42e7-8632-08a20c55545a\" (UID: \"bf69fbf0-3136-42e7-8632-08a20c55545a\") "
Sep 05 02:21:35.448347 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448319    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:35.448422 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448363    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:35.448482 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448446    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-oauth-serving-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:35.448482 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448466    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-trusted-ca-bundle\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:35.448594 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-console-config" (OuterVolumeSpecName: "console-config") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:35.448650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.448589    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-service-ca" (OuterVolumeSpecName: "service-ca") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:35.450195 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.450162    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf69fbf0-3136-42e7-8632-08a20c55545a-kube-api-access-5sk4l" (OuterVolumeSpecName: "kube-api-access-5sk4l") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "kube-api-access-5sk4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:35.450319 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.450201    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:35.450319 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.450238    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bf69fbf0-3136-42e7-8632-08a20c55545a" (UID: "bf69fbf0-3136-42e7-8632-08a20c55545a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:35.549309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.549275    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-oauth-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:35.549309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.549302    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf69fbf0-3136-42e7-8632-08a20c55545a-console-serving-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:35.549309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.549313    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sk4l\" (UniqueName: \"kubernetes.io/projected/bf69fbf0-3136-42e7-8632-08a20c55545a-kube-api-access-5sk4l\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:35.549309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.549322    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-console-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:35.549569 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:35.549330    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf69fbf0-3136-42e7-8632-08a20c55545a-service-ca\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:21:36.122947 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.122915    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf69fbf0-3136-42e7-8632-08a20c55545a" containerID="e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f" exitCode=0
Sep 05 02:21:36.123361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.122988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f58f69bb-7hjhv" event={"ID":"bf69fbf0-3136-42e7-8632-08a20c55545a","Type":"ContainerDied","Data":"e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f"}
Sep 05 02:21:36.123361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.122993    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f58f69bb-7hjhv"
Sep 05 02:21:36.123361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.123030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f58f69bb-7hjhv" event={"ID":"bf69fbf0-3136-42e7-8632-08a20c55545a","Type":"ContainerDied","Data":"eb1f312436519ec945bf50519423445ac616e02e2e084398fa89942d4e202df2"}
Sep 05 02:21:36.123361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.123050    2582 scope.go:117] "RemoveContainer" containerID="e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f"
Sep 05 02:21:36.131145 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.131127    2582 scope.go:117] "RemoveContainer" containerID="e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f"
Sep 05 02:21:36.131410 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:21:36.131391    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f\": container with ID starting with e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f not found: ID does not exist" containerID="e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f"
Sep 05 02:21:36.131448 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.131419    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f"} err="failed to get container status \"e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f\": rpc error: code = NotFound desc = could not find container \"e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f\": container with ID starting with e317edac0f655d3733067b45e6379bf3f4a7d209df9009efb2c7a7afeeb3e19f not found: ID does not exist"
Sep 05 02:21:36.139606 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.139581    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f58f69bb-7hjhv"]
Sep 05 02:21:36.143538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:36.143512    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f58f69bb-7hjhv"]
Sep 05 02:21:38.108446 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:21:38.108408    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf69fbf0-3136-42e7-8632-08a20c55545a" path="/var/lib/kubelet/pods/bf69fbf0-3136-42e7-8632-08a20c55545a/volumes"
Sep 05 02:22:40.015768 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:40.015740    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:22:52.968600 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:52.968568    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f4dd696dc-zkq9w"]
Sep 05 02:22:52.970864 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:52.968920    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf69fbf0-3136-42e7-8632-08a20c55545a" containerName="console"
Sep 05 02:22:52.970864 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:52.968934    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf69fbf0-3136-42e7-8632-08a20c55545a" containerName="console"
Sep 05 02:22:52.970864 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:52.969017    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf69fbf0-3136-42e7-8632-08a20c55545a" containerName="console"
Sep 05 02:22:52.971787 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:52.971770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:52.981464 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:52.981439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4dd696dc-zkq9w"]
Sep 05 02:22:53.087448 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-serving-cert\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.087448 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-oauth-serving-cert\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.087650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-console-config\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.087650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-oauth-config\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.087650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbk6\" (UniqueName: \"kubernetes.io/projected/2dab68de-021d-4d07-9cef-f127a3a11576-kube-api-access-msbk6\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.087650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-trusted-ca-bundle\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.087650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.087641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-service-ca\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.188920 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.188877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbk6\" (UniqueName: \"kubernetes.io/projected/2dab68de-021d-4d07-9cef-f127a3a11576-kube-api-access-msbk6\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.188920 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.188925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-trusted-ca-bundle\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.188962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-service-ca\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-serving-cert\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-oauth-serving-cert\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-console-config\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189396 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-oauth-config\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189829 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-service-ca\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.189829 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-console-config\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.190056 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-trusted-ca-bundle\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.190056 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.189925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-oauth-serving-cert\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.191652 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.191625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-serving-cert\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.191747 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.191723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-oauth-config\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.197116 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.197098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbk6\" (UniqueName: \"kubernetes.io/projected/2dab68de-021d-4d07-9cef-f127a3a11576-kube-api-access-msbk6\") pod \"console-6f4dd696dc-zkq9w\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") " pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.280687 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.280598    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:22:53.551934 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:53.551855    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4dd696dc-zkq9w"]
Sep 05 02:22:53.555082 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:22:53.555032    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dab68de_021d_4d07_9cef_f127a3a11576.slice/crio-8eed9ca9d65b79a4f6c233e616a0c1074d8dab37a3ef0fcdf10858ec0b72919a WatchSource:0}: Error finding container 8eed9ca9d65b79a4f6c233e616a0c1074d8dab37a3ef0fcdf10858ec0b72919a: Status 404 returned error can't find the container with id 8eed9ca9d65b79a4f6c233e616a0c1074d8dab37a3ef0fcdf10858ec0b72919a
Sep 05 02:22:54.428606 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:54.428574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4dd696dc-zkq9w" event={"ID":"2dab68de-021d-4d07-9cef-f127a3a11576","Type":"ContainerStarted","Data":"164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d"}
Sep 05 02:22:54.429014 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:54.428623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4dd696dc-zkq9w" event={"ID":"2dab68de-021d-4d07-9cef-f127a3a11576","Type":"ContainerStarted","Data":"8eed9ca9d65b79a4f6c233e616a0c1074d8dab37a3ef0fcdf10858ec0b72919a"}
Sep 05 02:22:54.448844 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:22:54.448796    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f4dd696dc-zkq9w" podStartSLOduration=2.448779291 podStartE2EDuration="2.448779291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:22:54.447722121 +0000 UTC m=+314.869227485" watchObservedRunningTime="2026-09-05 02:22:54.448779291 +0000 UTC m=+314.870284630"
Sep 05 02:23:03.281669 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:03.281624    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:23:03.282071 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:03.281684    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:23:03.286522 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:03.286498    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:23:03.468298 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:03.468269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:23:03.511800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:03.511769    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d54df6f9c-pvs7c"]
Sep 05 02:23:28.534351 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.534308    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d54df6f9c-pvs7c" podUID="01f07e51-c6cc-4221-a1cb-b3b866e0e842" containerName="console" containerID="cri-o://e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7" gracePeriod=15
Sep 05 02:23:28.776066 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.776041    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:23:28.886185 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886154    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-oauth-config\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886185 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886190    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-service-ca\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886420 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz7jb\" (UniqueName: \"kubernetes.io/projected/01f07e51-c6cc-4221-a1cb-b3b866e0e842-kube-api-access-fz7jb\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886420 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886269    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-config\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886420 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-serving-cert\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886420 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-oauth-serving-cert\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886420 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886372    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-trusted-ca-bundle\") pod \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\" (UID: \"01f07e51-c6cc-4221-a1cb-b3b866e0e842\") "
Sep 05 02:23:28.886710 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886680    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-service-ca" (OuterVolumeSpecName: "service-ca") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:28.886795 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886733    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-config" (OuterVolumeSpecName: "console-config") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:28.886795 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886756    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:28.886889 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.886856    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:23:28.888433 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.888413    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f07e51-c6cc-4221-a1cb-b3b866e0e842-kube-api-access-fz7jb" (OuterVolumeSpecName: "kube-api-access-fz7jb") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "kube-api-access-fz7jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:23:28.888845 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.888821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:23:28.888939 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.888846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01f07e51-c6cc-4221-a1cb-b3b866e0e842" (UID: "01f07e51-c6cc-4221-a1cb-b3b866e0e842"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:23:28.987328 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987291    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-oauth-serving-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:28.987328 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987319    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-trusted-ca-bundle\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:28.987328 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987329    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-oauth-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:28.987328 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987338    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-service-ca\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:28.987593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987346    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7jb\" (UniqueName: \"kubernetes.io/projected/01f07e51-c6cc-4221-a1cb-b3b866e0e842-kube-api-access-fz7jb\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:28.987593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987355    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:28.987593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:28.987364    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01f07e51-c6cc-4221-a1cb-b3b866e0e842-console-serving-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:23:29.572582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.572552    2582 generic.go:356] "Generic (PLEG): container finished" podID="01f07e51-c6cc-4221-a1cb-b3b866e0e842" containerID="e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7" exitCode=0
Sep 05 02:23:29.573000 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.572599    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d54df6f9c-pvs7c"
Sep 05 02:23:29.573000 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.572649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d54df6f9c-pvs7c" event={"ID":"01f07e51-c6cc-4221-a1cb-b3b866e0e842","Type":"ContainerDied","Data":"e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7"}
Sep 05 02:23:29.573000 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.572702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d54df6f9c-pvs7c" event={"ID":"01f07e51-c6cc-4221-a1cb-b3b866e0e842","Type":"ContainerDied","Data":"4083a5a2198aed2dc9b284a122eb8686f306e32e4ca5fd732d33cc1a129a577a"}
Sep 05 02:23:29.573000 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.572726    2582 scope.go:117] "RemoveContainer" containerID="e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7"
Sep 05 02:23:29.581387 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.581367    2582 scope.go:117] "RemoveContainer" containerID="e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7"
Sep 05 02:23:29.581666 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:23:29.581648    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7\": container with ID starting with e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7 not found: ID does not exist" containerID="e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7"
Sep 05 02:23:29.581712 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.581676    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7"} err="failed to get container status \"e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7\": rpc error: code = NotFound desc = could not find container \"e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7\": container with ID starting with e9cafc60df8723530359fa822eb8aaa044a6b9c4de471d15d6b1f550759332f7 not found: ID does not exist"
Sep 05 02:23:29.593140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.593114    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d54df6f9c-pvs7c"]
Sep 05 02:23:29.598416 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:29.598392    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d54df6f9c-pvs7c"]
Sep 05 02:23:30.109155 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:23:30.109122    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f07e51-c6cc-4221-a1cb-b3b866e0e842" path="/var/lib/kubelet/pods/01f07e51-c6cc-4221-a1cb-b3b866e0e842/volumes"
Sep 05 02:24:05.493924 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.493888    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"]
Sep 05 02:24:05.494407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.494218    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f07e51-c6cc-4221-a1cb-b3b866e0e842" containerName="console"
Sep 05 02:24:05.494407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.494230    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f07e51-c6cc-4221-a1cb-b3b866e0e842" containerName="console"
Sep 05 02:24:05.494407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.494298    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f07e51-c6cc-4221-a1cb-b3b866e0e842" containerName="console"
Sep 05 02:24:05.497321 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.497301    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.499178 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.499152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.499303 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.499216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/dee34980-b302-41be-983b-008e45f7f85c-kube-api-access-tmv89\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.499303 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.499254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.500410 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.500393    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:24:05.501650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.501631    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:24:05.501743 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.501656    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gbq6r\""
Sep 05 02:24:05.509608 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.509573    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"]
Sep 05 02:24:05.600309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.600276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/dee34980-b302-41be-983b-008e45f7f85c-kube-api-access-tmv89\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.600309 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.600318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.600535 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.600382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.600711 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.600695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.600749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.600737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.609599 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.609559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/dee34980-b302-41be-983b-008e45f7f85c-kube-api-access-tmv89\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:05.806015 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:05.805911    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:06.076292 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:06.076258    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"]
Sep 05 02:24:06.080197 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:24:06.080166    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee34980_b302_41be_983b_008e45f7f85c.slice/crio-6e645d8a38f82f8bc2181c7cfba85b39ee26d799787330ed8e973f26d58cf754 WatchSource:0}: Error finding container 6e645d8a38f82f8bc2181c7cfba85b39ee26d799787330ed8e973f26d58cf754: Status 404 returned error can't find the container with id 6e645d8a38f82f8bc2181c7cfba85b39ee26d799787330ed8e973f26d58cf754
Sep 05 02:24:06.082213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:06.082197    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:24:06.715604 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:06.715559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb" event={"ID":"dee34980-b302-41be-983b-008e45f7f85c","Type":"ContainerStarted","Data":"6e645d8a38f82f8bc2181c7cfba85b39ee26d799787330ed8e973f26d58cf754"}
Sep 05 02:24:12.745595 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:12.745553    2582 generic.go:356] "Generic (PLEG): container finished" podID="dee34980-b302-41be-983b-008e45f7f85c" containerID="92f9cf6aa72f2e152f3e5e73ea0def3e4a043704d8fd354b88e61ba6d8161102" exitCode=0
Sep 05 02:24:12.746009 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:12.745645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb" event={"ID":"dee34980-b302-41be-983b-008e45f7f85c","Type":"ContainerDied","Data":"92f9cf6aa72f2e152f3e5e73ea0def3e4a043704d8fd354b88e61ba6d8161102"}
Sep 05 02:24:15.759259 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:15.759224    2582 generic.go:356] "Generic (PLEG): container finished" podID="dee34980-b302-41be-983b-008e45f7f85c" containerID="608e6dfcc838ea1b59b508759241715ec8d659d249050c13be5f2dc9d66c184f" exitCode=0
Sep 05 02:24:15.759683 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:15.759277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb" event={"ID":"dee34980-b302-41be-983b-008e45f7f85c","Type":"ContainerDied","Data":"608e6dfcc838ea1b59b508759241715ec8d659d249050c13be5f2dc9d66c184f"}
Sep 05 02:24:22.796772 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:22.796735    2582 generic.go:356] "Generic (PLEG): container finished" podID="dee34980-b302-41be-983b-008e45f7f85c" containerID="eb1a953aa256a498e9a54ddcd389a3d3fbc486ad59e0344d53cac9c6eec6c791" exitCode=0
Sep 05 02:24:22.797220 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:22.796825    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb" event={"ID":"dee34980-b302-41be-983b-008e45f7f85c","Type":"ContainerDied","Data":"eb1a953aa256a498e9a54ddcd389a3d3fbc486ad59e0344d53cac9c6eec6c791"}
Sep 05 02:24:23.922761 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:23.922734    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:24.058218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.058119    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/dee34980-b302-41be-983b-008e45f7f85c-kube-api-access-tmv89\") pod \"dee34980-b302-41be-983b-008e45f7f85c\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") "
Sep 05 02:24:24.058218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.058170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-util\") pod \"dee34980-b302-41be-983b-008e45f7f85c\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") "
Sep 05 02:24:24.058218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.058228    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-bundle\") pod \"dee34980-b302-41be-983b-008e45f7f85c\" (UID: \"dee34980-b302-41be-983b-008e45f7f85c\") "
Sep 05 02:24:24.058906 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.058876    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-bundle" (OuterVolumeSpecName: "bundle") pod "dee34980-b302-41be-983b-008e45f7f85c" (UID: "dee34980-b302-41be-983b-008e45f7f85c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:24:24.060390 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.060367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee34980-b302-41be-983b-008e45f7f85c-kube-api-access-tmv89" (OuterVolumeSpecName: "kube-api-access-tmv89") pod "dee34980-b302-41be-983b-008e45f7f85c" (UID: "dee34980-b302-41be-983b-008e45f7f85c"). InnerVolumeSpecName "kube-api-access-tmv89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:24:24.062165 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.062143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-util" (OuterVolumeSpecName: "util") pod "dee34980-b302-41be-983b-008e45f7f85c" (UID: "dee34980-b302-41be-983b-008e45f7f85c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:24:24.159362 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.159327    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-bundle\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:24:24.159362 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.159358    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmv89\" (UniqueName: \"kubernetes.io/projected/dee34980-b302-41be-983b-008e45f7f85c-kube-api-access-tmv89\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:24:24.159362 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.159368    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dee34980-b302-41be-983b-008e45f7f85c-util\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:24:24.806952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.806921    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb"
Sep 05 02:24:24.807134 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.806930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a84hjb" event={"ID":"dee34980-b302-41be-983b-008e45f7f85c","Type":"ContainerDied","Data":"6e645d8a38f82f8bc2181c7cfba85b39ee26d799787330ed8e973f26d58cf754"}
Sep 05 02:24:24.807134 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:24.807016    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e645d8a38f82f8bc2181c7cfba85b39ee26d799787330ed8e973f26d58cf754"
Sep 05 02:24:33.069530 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.069490    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4t8gn"]
Sep 05 02:24:33.070023 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.069992    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="pull"
Sep 05 02:24:33.070023 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.070011    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="pull"
Sep 05 02:24:33.070023 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.070023    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="extract"
Sep 05 02:24:33.070194 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.070032    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="extract"
Sep 05 02:24:33.070194 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.070045    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="util"
Sep 05 02:24:33.070194 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.070054    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="util"
Sep 05 02:24:33.070194 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.070143    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee34980-b302-41be-983b-008e45f7f85c" containerName="extract"
Sep 05 02:24:33.073085 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.073065    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.075626 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.075599    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5hl79\""
Sep 05 02:24:33.075626 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.075612    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 05 02:24:33.075796 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.075636    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 05 02:24:33.075796 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.075669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 05 02:24:33.076489 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.076474    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 05 02:24:33.076489 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.076485    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 05 02:24:33.082707 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.082688    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4t8gn"]
Sep 05 02:24:33.134411 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.134376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-cabundle0\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.134593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.134430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8sv6\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-kube-api-access-c8sv6\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.134593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.134459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-certificates\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.235807 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.235770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-cabundle0\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.235959 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.235820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8sv6\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-kube-api-access-c8sv6\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.236112 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.236084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-certificates\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.236222 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.236204    2582 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 05 02:24:33.236286 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.236228    2582 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:24:33.236286 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.236240    2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:24:33.236286 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.236254    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-4t8gn: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 05 02:24:33.236441 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.236304    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-certificates podName:47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1 nodeName:}" failed. No retries permitted until 2026-09-05 02:24:33.736287207 +0000 UTC m=+414.157792532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-certificates") pod "keda-operator-579c8958f7-4t8gn" (UID: "47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 05 02:24:33.236610 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.236592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-cabundle0\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.246075 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.246046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8sv6\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-kube-api-access-c8sv6\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.286684 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.286647    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"]
Sep 05 02:24:33.290072 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.290051    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.293203 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.293181    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 05 02:24:33.304245 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.304218    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"]
Sep 05 02:24:33.437213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.437123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c88e8965-8610-43da-a42d-fc26f35673ec-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.437213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.437177    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.437213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.437205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t22l\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-kube-api-access-9t22l\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.538064 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.538011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.538064 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.538066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t22l\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-kube-api-access-9t22l\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.538399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.538145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c88e8965-8610-43da-a42d-fc26f35673ec-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.538399 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.538166    2582 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:33.538399 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.538194    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp: secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:33.538399 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:33.538243    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates podName:c88e8965-8610-43da-a42d-fc26f35673ec nodeName:}" failed. No retries permitted until 2026-09-05 02:24:34.038228607 +0000 UTC m=+414.459733923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates") pod "keda-metrics-apiserver-77857b7b98-rdpzp" (UID: "c88e8965-8610-43da-a42d-fc26f35673ec") : secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:33.538590 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.538494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c88e8965-8610-43da-a42d-fc26f35673ec-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.549220 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.549197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t22l\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-kube-api-access-9t22l\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:33.620150 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.620115    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-brpwz"]
Sep 05 02:24:33.623539 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.623523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.629461 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.629435    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 05 02:24:33.636081 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.636057    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-brpwz"]
Sep 05 02:24:33.740349 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.740261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6csmb\" (UniqueName: \"kubernetes.io/projected/0f554432-dd42-4dbd-8551-8fb1a6058eed-kube-api-access-6csmb\") pod \"keda-admission-6c4d864b86-brpwz\" (UID: \"0f554432-dd42-4dbd-8551-8fb1a6058eed\") " pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.740349 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.740341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f554432-dd42-4dbd-8551-8fb1a6058eed-certificates\") pod \"keda-admission-6c4d864b86-brpwz\" (UID: \"0f554432-dd42-4dbd-8551-8fb1a6058eed\") " pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.740549 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.740402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-certificates\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.742788 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.742764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1-certificates\") pod \"keda-operator-579c8958f7-4t8gn\" (UID: \"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1\") " pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:33.841529 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.841496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6csmb\" (UniqueName: \"kubernetes.io/projected/0f554432-dd42-4dbd-8551-8fb1a6058eed-kube-api-access-6csmb\") pod \"keda-admission-6c4d864b86-brpwz\" (UID: \"0f554432-dd42-4dbd-8551-8fb1a6058eed\") " pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.841699 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.841676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f554432-dd42-4dbd-8551-8fb1a6058eed-certificates\") pod \"keda-admission-6c4d864b86-brpwz\" (UID: \"0f554432-dd42-4dbd-8551-8fb1a6058eed\") " pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.844116 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.844098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f554432-dd42-4dbd-8551-8fb1a6058eed-certificates\") pod \"keda-admission-6c4d864b86-brpwz\" (UID: \"0f554432-dd42-4dbd-8551-8fb1a6058eed\") " pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.852205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.852185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6csmb\" (UniqueName: \"kubernetes.io/projected/0f554432-dd42-4dbd-8551-8fb1a6058eed-kube-api-access-6csmb\") pod \"keda-admission-6c4d864b86-brpwz\" (UID: \"0f554432-dd42-4dbd-8551-8fb1a6058eed\") " pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.934192 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.934152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:33.985500 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:33.985470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:34.043030 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:34.042938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:34.043377 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:34.043197    2582 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:34.043377 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:34.043223    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp: secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:34.043377 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:24:34.043283    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates podName:c88e8965-8610-43da-a42d-fc26f35673ec nodeName:}" failed. No retries permitted until 2026-09-05 02:24:35.043262135 +0000 UTC m=+415.464767475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates") pod "keda-metrics-apiserver-77857b7b98-rdpzp" (UID: "c88e8965-8610-43da-a42d-fc26f35673ec") : secret "keda-metrics-apiserver-certs" not found
Sep 05 02:24:34.206359 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:34.206329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-brpwz"]
Sep 05 02:24:34.209586 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:24:34.209562    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f554432_dd42_4dbd_8551_8fb1a6058eed.slice/crio-4953d1c0d5aacf0098ef307d480d6c3a669eab506d51a3d7e25aa5608f8073a5 WatchSource:0}: Error finding container 4953d1c0d5aacf0098ef307d480d6c3a669eab506d51a3d7e25aa5608f8073a5: Status 404 returned error can't find the container with id 4953d1c0d5aacf0098ef307d480d6c3a669eab506d51a3d7e25aa5608f8073a5
Sep 05 02:24:34.271486 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:34.271458    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4t8gn"]
Sep 05 02:24:34.272899 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:24:34.272876    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47457e5f_cdf3_4bdc_ae6c_9e2d2adc81d1.slice/crio-5a37c4b26ebda6af4de2e9a6e7ee557e20ca3a7d201449445aade86990af15fa WatchSource:0}: Error finding container 5a37c4b26ebda6af4de2e9a6e7ee557e20ca3a7d201449445aade86990af15fa: Status 404 returned error can't find the container with id 5a37c4b26ebda6af4de2e9a6e7ee557e20ca3a7d201449445aade86990af15fa
Sep 05 02:24:34.847778 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:34.847738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-4t8gn" event={"ID":"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1","Type":"ContainerStarted","Data":"5a37c4b26ebda6af4de2e9a6e7ee557e20ca3a7d201449445aade86990af15fa"}
Sep 05 02:24:34.848807 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:34.848782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-brpwz" event={"ID":"0f554432-dd42-4dbd-8551-8fb1a6058eed","Type":"ContainerStarted","Data":"4953d1c0d5aacf0098ef307d480d6c3a669eab506d51a3d7e25aa5608f8073a5"}
Sep 05 02:24:35.053269 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:35.053115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:35.056328 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:35.056284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c88e8965-8610-43da-a42d-fc26f35673ec-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rdpzp\" (UID: \"c88e8965-8610-43da-a42d-fc26f35673ec\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:35.100353 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:35.100248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:35.397792 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:35.397762    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"]
Sep 05 02:24:35.401092 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:24:35.401053    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88e8965_8610_43da_a42d_fc26f35673ec.slice/crio-d75f0de8942fc654c707e760b9e13efcda543b614def1f3b94bef788912df9a2 WatchSource:0}: Error finding container d75f0de8942fc654c707e760b9e13efcda543b614def1f3b94bef788912df9a2: Status 404 returned error can't find the container with id d75f0de8942fc654c707e760b9e13efcda543b614def1f3b94bef788912df9a2
Sep 05 02:24:35.855655 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:35.855619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp" event={"ID":"c88e8965-8610-43da-a42d-fc26f35673ec","Type":"ContainerStarted","Data":"d75f0de8942fc654c707e760b9e13efcda543b614def1f3b94bef788912df9a2"}
Sep 05 02:24:36.861598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:36.861550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-brpwz" event={"ID":"0f554432-dd42-4dbd-8551-8fb1a6058eed","Type":"ContainerStarted","Data":"e24a885e8f3f1b73e7187e46679aa306060973fe8d7825eeb5c19403e5798ba9"}
Sep 05 02:24:36.862107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:36.861672    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:36.891578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:36.891516    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-brpwz" podStartSLOduration=2.266750026 podStartE2EDuration="3.891493736s" totalImagesPullingTime="1.62474371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:36.889635809 +0000 UTC m=+417.311141150" watchObservedRunningTime="2026-09-05 02:24:36.891493736 +0000 UTC m=+417.312999078"
Sep 05 02:24:38.872099 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:38.872061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-4t8gn" event={"ID":"47457e5f-cdf3-4bdc-ae6c-9e2d2adc81d1","Type":"ContainerStarted","Data":"74d1248fbe706d711d24587ac4fef8ffe178551808fafe06d8b697618ffb48a5"}
Sep 05 02:24:38.872527 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:38.872198    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:24:38.873827 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:38.873803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp" event={"ID":"c88e8965-8610-43da-a42d-fc26f35673ec","Type":"ContainerStarted","Data":"dda71dea04cbe35d7d82fa336e0dc5ed8fd59308ed8b82692f88a2b42f720edd"}
Sep 05 02:24:38.873984 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:38.873891    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:38.892708 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:38.892665    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-4t8gn" podStartSLOduration=1.522938404 podStartE2EDuration="5.892649298s" totalImagesPullingTime="4.369710894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:38.891236805 +0000 UTC m=+419.312742146" watchObservedRunningTime="2026-09-05 02:24:38.892649298 +0000 UTC m=+419.314154637"
Sep 05 02:24:38.915497 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:38.915386    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp" podStartSLOduration=2.673255154 podStartE2EDuration="5.915364335s" totalImagesPullingTime="3.242109181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:38.914362041 +0000 UTC m=+419.335867381" watchObservedRunningTime="2026-09-05 02:24:38.915364335 +0000 UTC m=+419.336869675"
Sep 05 02:24:49.882934 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:49.882905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rdpzp"
Sep 05 02:24:57.869375 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:57.869345    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-brpwz"
Sep 05 02:24:59.880555 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:24:59.880524    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-4t8gn"
Sep 05 02:25:53.647076 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.647039    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5788fcf757-hczxm"]
Sep 05 02:25:53.649504 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.649486    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.671204 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.671176    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5788fcf757-hczxm"]
Sep 05 02:25:53.737672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-service-ca\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.737672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxtgb\" (UniqueName: \"kubernetes.io/projected/7ac71966-dc18-41de-8dcd-914f7002bf52-kube-api-access-dxtgb\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.737883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-oauth-serving-cert\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.737883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ac71966-dc18-41de-8dcd-914f7002bf52-console-serving-cert\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.737883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ac71966-dc18-41de-8dcd-914f7002bf52-console-oauth-config\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.738004 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-trusted-ca-bundle\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.738004 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.737910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-console-config\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839156 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-trusted-ca-bundle\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839156 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-console-config\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-service-ca\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxtgb\" (UniqueName: \"kubernetes.io/projected/7ac71966-dc18-41de-8dcd-914f7002bf52-kube-api-access-dxtgb\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-oauth-serving-cert\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839407 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ac71966-dc18-41de-8dcd-914f7002bf52-console-serving-cert\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.839604 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.839421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ac71966-dc18-41de-8dcd-914f7002bf52-console-oauth-config\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.840115 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.840086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-service-ca\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.840249 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.840146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-oauth-serving-cert\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.840249 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.840162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-trusted-ca-bundle\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.840249 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.840177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ac71966-dc18-41de-8dcd-914f7002bf52-console-config\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.841761 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.841733    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ac71966-dc18-41de-8dcd-914f7002bf52-console-serving-cert\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.841861 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.841838    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ac71966-dc18-41de-8dcd-914f7002bf52-console-oauth-config\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.853113 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.853089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxtgb\" (UniqueName: \"kubernetes.io/projected/7ac71966-dc18-41de-8dcd-914f7002bf52-kube-api-access-dxtgb\") pod \"console-5788fcf757-hczxm\" (UID: \"7ac71966-dc18-41de-8dcd-914f7002bf52\") " pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:53.957809 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:53.957721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:25:54.284124 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:54.284098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5788fcf757-hczxm"]
Sep 05 02:25:54.286008 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:25:54.285983    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac71966_dc18_41de_8dcd_914f7002bf52.slice/crio-8db16b0f85d9190fde6f592e3c345e787a1706916d5b8a086d3c10b1d5263303 WatchSource:0}: Error finding container 8db16b0f85d9190fde6f592e3c345e787a1706916d5b8a086d3c10b1d5263303: Status 404 returned error can't find the container with id 8db16b0f85d9190fde6f592e3c345e787a1706916d5b8a086d3c10b1d5263303
Sep 05 02:25:55.189677 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:55.189640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5788fcf757-hczxm" event={"ID":"7ac71966-dc18-41de-8dcd-914f7002bf52","Type":"ContainerStarted","Data":"ba717b54dc258dfe36f5ec2f2539975ff12eee454c8bf4a9ad9d57ad16e59168"}
Sep 05 02:25:55.190201 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:55.189700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5788fcf757-hczxm" event={"ID":"7ac71966-dc18-41de-8dcd-914f7002bf52","Type":"ContainerStarted","Data":"8db16b0f85d9190fde6f592e3c345e787a1706916d5b8a086d3c10b1d5263303"}
Sep 05 02:25:55.226410 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:25:55.226359    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5788fcf757-hczxm" podStartSLOduration=2.226341149 podStartE2EDuration="2.226341149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:55.226054095 +0000 UTC m=+495.647559435" watchObservedRunningTime="2026-09-05 02:25:55.226341149 +0000 UTC m=+495.647846490"
Sep 05 02:26:03.510540 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.510508    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-c287p"]
Sep 05 02:26:03.512802 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.512780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.515411 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.515387    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:26:03.515526 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.515447    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:26:03.516382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.516365    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 05 02:26:03.516463 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.516381    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7mkrw\""
Sep 05 02:26:03.523108 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.523087    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-c287p"]
Sep 05 02:26:03.623609 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.623570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvmb\" (UniqueName: \"kubernetes.io/projected/196c9abe-1bf7-492d-8794-fdf8dfa847fe-kube-api-access-mtvmb\") pod \"seaweedfs-64568bcd49-c287p\" (UID: \"196c9abe-1bf7-492d-8794-fdf8dfa847fe\") " pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.623785 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.623632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/196c9abe-1bf7-492d-8794-fdf8dfa847fe-data\") pod \"seaweedfs-64568bcd49-c287p\" (UID: \"196c9abe-1bf7-492d-8794-fdf8dfa847fe\") " pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.724521 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.724482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvmb\" (UniqueName: \"kubernetes.io/projected/196c9abe-1bf7-492d-8794-fdf8dfa847fe-kube-api-access-mtvmb\") pod \"seaweedfs-64568bcd49-c287p\" (UID: \"196c9abe-1bf7-492d-8794-fdf8dfa847fe\") " pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.724694 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.724538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/196c9abe-1bf7-492d-8794-fdf8dfa847fe-data\") pod \"seaweedfs-64568bcd49-c287p\" (UID: \"196c9abe-1bf7-492d-8794-fdf8dfa847fe\") " pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.724948 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.724929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/196c9abe-1bf7-492d-8794-fdf8dfa847fe-data\") pod \"seaweedfs-64568bcd49-c287p\" (UID: \"196c9abe-1bf7-492d-8794-fdf8dfa847fe\") " pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.732954 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.732935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvmb\" (UniqueName: \"kubernetes.io/projected/196c9abe-1bf7-492d-8794-fdf8dfa847fe-kube-api-access-mtvmb\") pod \"seaweedfs-64568bcd49-c287p\" (UID: \"196c9abe-1bf7-492d-8794-fdf8dfa847fe\") " pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.822741 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.822708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:03.957859 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.957818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:26:03.958082 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.957890    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:26:03.962507 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:03.962482    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:26:04.094394 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:04.094369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-c287p"]
Sep 05 02:26:04.097064 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:26:04.097031    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196c9abe_1bf7_492d_8794_fdf8dfa847fe.slice/crio-7eaa441a2d27ba42ce2844ac45e87fa45a719434bec5e5b9aa28e8e9c01ee983 WatchSource:0}: Error finding container 7eaa441a2d27ba42ce2844ac45e87fa45a719434bec5e5b9aa28e8e9c01ee983: Status 404 returned error can't find the container with id 7eaa441a2d27ba42ce2844ac45e87fa45a719434bec5e5b9aa28e8e9c01ee983
Sep 05 02:26:04.232446 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:04.232410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-c287p" event={"ID":"196c9abe-1bf7-492d-8794-fdf8dfa847fe","Type":"ContainerStarted","Data":"7eaa441a2d27ba42ce2844ac45e87fa45a719434bec5e5b9aa28e8e9c01ee983"}
Sep 05 02:26:04.236392 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:04.236365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5788fcf757-hczxm"
Sep 05 02:26:04.284373 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:04.284338    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4dd696dc-zkq9w"]
Sep 05 02:26:07.248578 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:07.248541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-c287p" event={"ID":"196c9abe-1bf7-492d-8794-fdf8dfa847fe","Type":"ContainerStarted","Data":"a1944985f46bd92b06e719507c8e758e9cc642def88bba552195ba0ba396091b"}
Sep 05 02:26:07.248952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:07.248716    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:07.266846 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:07.266795    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-c287p" podStartSLOduration=1.25947481 podStartE2EDuration="4.266780129s" totalImagesPullingTime="3.007305319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:07.265487052 +0000 UTC m=+507.686992403" watchObservedRunningTime="2026-09-05 02:26:07.266780129 +0000 UTC m=+507.688285489"
Sep 05 02:26:13.256061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:13.256025    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-c287p"
Sep 05 02:26:29.307921 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.307879    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f4dd696dc-zkq9w" podUID="2dab68de-021d-4d07-9cef-f127a3a11576" containerName="console" containerID="cri-o://164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d" gracePeriod=15
Sep 05 02:26:29.566878 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.566850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:26:29.640242 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-serving-cert\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640261    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-trusted-ca-bundle\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640298    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msbk6\" (UniqueName: \"kubernetes.io/projected/2dab68de-021d-4d07-9cef-f127a3a11576-kube-api-access-msbk6\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-service-ca\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640365    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-console-config\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-oauth-config\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640678 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640434    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-oauth-serving-cert\") pod \"2dab68de-021d-4d07-9cef-f127a3a11576\" (UID: \"2dab68de-021d-4d07-9cef-f127a3a11576\") "
Sep 05 02:26:29.640770 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640737    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.640880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-service-ca" (OuterVolumeSpecName: "service-ca") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.640880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.640802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-console-config" (OuterVolumeSpecName: "console-config") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.641148 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.641115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:26:29.642606 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.642582    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:26:29.642883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.642858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dab68de-021d-4d07-9cef-f127a3a11576-kube-api-access-msbk6" (OuterVolumeSpecName: "kube-api-access-msbk6") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "kube-api-access-msbk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:29.642883 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.642873    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2dab68de-021d-4d07-9cef-f127a3a11576" (UID: "2dab68de-021d-4d07-9cef-f127a3a11576"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:26:29.741506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741458    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-serving-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.741506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741502    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-trusted-ca-bundle\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.741506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741512    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbk6\" (UniqueName: \"kubernetes.io/projected/2dab68de-021d-4d07-9cef-f127a3a11576-kube-api-access-msbk6\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.741506 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741525    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-service-ca\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.741791 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741534    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-console-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.741791 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741542    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dab68de-021d-4d07-9cef-f127a3a11576-console-oauth-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:29.741791 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:29.741550    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dab68de-021d-4d07-9cef-f127a3a11576-oauth-serving-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:26:30.345426 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.345387    2582 generic.go:356] "Generic (PLEG): container finished" podID="2dab68de-021d-4d07-9cef-f127a3a11576" containerID="164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d" exitCode=0
Sep 05 02:26:30.345923 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.345450    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4dd696dc-zkq9w"
Sep 05 02:26:30.345923 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.345476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4dd696dc-zkq9w" event={"ID":"2dab68de-021d-4d07-9cef-f127a3a11576","Type":"ContainerDied","Data":"164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d"}
Sep 05 02:26:30.345923 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.345541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4dd696dc-zkq9w" event={"ID":"2dab68de-021d-4d07-9cef-f127a3a11576","Type":"ContainerDied","Data":"8eed9ca9d65b79a4f6c233e616a0c1074d8dab37a3ef0fcdf10858ec0b72919a"}
Sep 05 02:26:30.345923 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.345577    2582 scope.go:117] "RemoveContainer" containerID="164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d"
Sep 05 02:26:30.354079 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.354059    2582 scope.go:117] "RemoveContainer" containerID="164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d"
Sep 05 02:26:30.354329 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:26:30.354311    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d\": container with ID starting with 164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d not found: ID does not exist" containerID="164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d"
Sep 05 02:26:30.354389 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.354338    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d"} err="failed to get container status \"164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d\": rpc error: code = NotFound desc = could not find container \"164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d\": container with ID starting with 164993911f45459a21abb0772f2b2be3108de4f934352e6914fe9f9cd227628d not found: ID does not exist"
Sep 05 02:26:30.363183 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.363160    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4dd696dc-zkq9w"]
Sep 05 02:26:30.369038 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:30.369015    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f4dd696dc-zkq9w"]
Sep 05 02:26:32.109669 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:26:32.109639    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dab68de-021d-4d07-9cef-f127a3a11576" path="/var/lib/kubelet/pods/2dab68de-021d-4d07-9cef-f127a3a11576/volumes"
Sep 05 02:27:21.313337 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.313305    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-tvwhg"]
Sep 05 02:27:21.313765 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.313714    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dab68de-021d-4d07-9cef-f127a3a11576" containerName="console"
Sep 05 02:27:21.313765 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.313727    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dab68de-021d-4d07-9cef-f127a3a11576" containerName="console"
Sep 05 02:27:21.313839 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.313778    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dab68de-021d-4d07-9cef-f127a3a11576" containerName="console"
Sep 05 02:27:21.316269 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.316255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.319852 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.319833    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 05 02:27:21.319852 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.319845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8vxmj\""
Sep 05 02:27:21.330361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.330336    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-zd6vp"]
Sep 05 02:27:21.333031 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.333010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-tvwhg"]
Sep 05 02:27:21.333142 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.333100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.335649 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.335629    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 05 02:27:21.335757 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.335633    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 05 02:27:21.335950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.335934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-f95nq\""
Sep 05 02:27:21.355850 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.355824    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-zd6vp"]
Sep 05 02:27:21.476548 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.476510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ba3b-9adb-44a9-9545-f72625fd5a40-tls-certs\") pod \"model-serving-api-7b954ffc95-tvwhg\" (UID: \"0409ba3b-9adb-44a9-9545-f72625fd5a40\") " pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.476731 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.476571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-cert\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.476731 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.476673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-metrics-certs\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.476850 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.476740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cjf4\" (UniqueName: \"kubernetes.io/projected/c71f4124-85ed-4953-93af-19433318a583-kube-api-access-6cjf4\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.476850 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.476759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9zp\" (UniqueName: \"kubernetes.io/projected/0409ba3b-9adb-44a9-9545-f72625fd5a40-kube-api-access-9z9zp\") pod \"model-serving-api-7b954ffc95-tvwhg\" (UID: \"0409ba3b-9adb-44a9-9545-f72625fd5a40\") " pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.578131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.578044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ba3b-9adb-44a9-9545-f72625fd5a40-tls-certs\") pod \"model-serving-api-7b954ffc95-tvwhg\" (UID: \"0409ba3b-9adb-44a9-9545-f72625fd5a40\") " pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.578131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.578098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-cert\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.578336 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.578142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-metrics-certs\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.578336 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.578203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cjf4\" (UniqueName: \"kubernetes.io/projected/c71f4124-85ed-4953-93af-19433318a583-kube-api-access-6cjf4\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.578336 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.578228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9zp\" (UniqueName: \"kubernetes.io/projected/0409ba3b-9adb-44a9-9545-f72625fd5a40-kube-api-access-9z9zp\") pod \"model-serving-api-7b954ffc95-tvwhg\" (UID: \"0409ba3b-9adb-44a9-9545-f72625fd5a40\") " pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.578484 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:27:21.578353    2582 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 05 02:27:21.578484 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:27:21.578429    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-metrics-certs podName:c71f4124-85ed-4953-93af-19433318a583 nodeName:}" failed. No retries permitted until 2026-09-05 02:27:22.078409194 +0000 UTC m=+582.499914525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-metrics-certs") pod "odh-model-controller-7b966d7984-zd6vp" (UID: "c71f4124-85ed-4953-93af-19433318a583") : secret "odh-model-controller-metrics-tls" not found
Sep 05 02:27:21.580806 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.580781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ba3b-9adb-44a9-9545-f72625fd5a40-tls-certs\") pod \"model-serving-api-7b954ffc95-tvwhg\" (UID: \"0409ba3b-9adb-44a9-9545-f72625fd5a40\") " pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.580880 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.580781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-cert\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.590005 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.589957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9zp\" (UniqueName: \"kubernetes.io/projected/0409ba3b-9adb-44a9-9545-f72625fd5a40-kube-api-access-9z9zp\") pod \"model-serving-api-7b954ffc95-tvwhg\" (UID: \"0409ba3b-9adb-44a9-9545-f72625fd5a40\") " pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.590339 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.590323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cjf4\" (UniqueName: \"kubernetes.io/projected/c71f4124-85ed-4953-93af-19433318a583-kube-api-access-6cjf4\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:21.626936 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.626907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:21.904930 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:21.904855    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-tvwhg"]
Sep 05 02:27:21.908684 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:27:21.908660    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0409ba3b_9adb_44a9_9545_f72625fd5a40.slice/crio-924e59dd65691b74c112e2c6e1981e44fabe968700f1a7f8d35a34ddc70e0edb WatchSource:0}: Error finding container 924e59dd65691b74c112e2c6e1981e44fabe968700f1a7f8d35a34ddc70e0edb: Status 404 returned error can't find the container with id 924e59dd65691b74c112e2c6e1981e44fabe968700f1a7f8d35a34ddc70e0edb
Sep 05 02:27:22.083558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:22.083517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-metrics-certs\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:22.085846 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:22.085814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71f4124-85ed-4953-93af-19433318a583-metrics-certs\") pod \"odh-model-controller-7b966d7984-zd6vp\" (UID: \"c71f4124-85ed-4953-93af-19433318a583\") " pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:22.245763 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:22.245673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:22.518514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:22.518470    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-zd6vp"]
Sep 05 02:27:22.521731 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:27:22.521700    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71f4124_85ed_4953_93af_19433318a583.slice/crio-17daf41b8e7fd9f3af887cb02a18191f7bbf64d385d0210ee3db2c65c1e60980 WatchSource:0}: Error finding container 17daf41b8e7fd9f3af887cb02a18191f7bbf64d385d0210ee3db2c65c1e60980: Status 404 returned error can't find the container with id 17daf41b8e7fd9f3af887cb02a18191f7bbf64d385d0210ee3db2c65c1e60980
Sep 05 02:27:22.561454 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:22.561373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-zd6vp" event={"ID":"c71f4124-85ed-4953-93af-19433318a583","Type":"ContainerStarted","Data":"17daf41b8e7fd9f3af887cb02a18191f7bbf64d385d0210ee3db2c65c1e60980"}
Sep 05 02:27:22.563162 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:22.563121    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-tvwhg" event={"ID":"0409ba3b-9adb-44a9-9545-f72625fd5a40","Type":"ContainerStarted","Data":"924e59dd65691b74c112e2c6e1981e44fabe968700f1a7f8d35a34ddc70e0edb"}
Sep 05 02:27:23.571227 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:23.570930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-tvwhg" event={"ID":"0409ba3b-9adb-44a9-9545-f72625fd5a40","Type":"ContainerStarted","Data":"4b8843b2a153f61a719082ab07e9dafd9e860ed7a595e5d17a8d7cf6f94e37ac"}
Sep 05 02:27:23.571658 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:23.571230    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:23.591866 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:23.591790    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-tvwhg" podStartSLOduration=1.29189892 podStartE2EDuration="2.591766526s" totalImagesPullingTime="1.299867606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:23.58857371 +0000 UTC m=+584.010079050" watchObservedRunningTime="2026-09-05 02:27:23.591766526 +0000 UTC m=+584.013271868"
Sep 05 02:27:25.581299 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:25.581264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-zd6vp" event={"ID":"c71f4124-85ed-4953-93af-19433318a583","Type":"ContainerStarted","Data":"d84424d638fb34c4ef44bcbd3cd4245617a857f34913f22e9e878c455dd3be08"}
Sep 05 02:27:25.581674 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:25.581398    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:25.600399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:25.600293    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-zd6vp" podStartSLOduration=1.789204012 podStartE2EDuration="4.600277252s" totalImagesPullingTime="2.81107324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:25.599107797 +0000 UTC m=+586.020613138" watchObservedRunningTime="2026-09-05 02:27:25.600277252 +0000 UTC m=+586.021782593"
Sep 05 02:27:34.580051 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:34.580019    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-tvwhg"
Sep 05 02:27:36.588208 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:36.588173    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-zd6vp"
Sep 05 02:27:37.475296 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.475257    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-t26fm"]
Sep 05 02:27:37.477624 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.477600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.480365 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.480344    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 05 02:27:37.491083 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.491062    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t26fm"]
Sep 05 02:27:37.518694 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.518664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e01e05-076f-41a9-8464-4176ceeb4d7d-model-cache\") pod \"s3-init-t26fm\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") " pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.518854 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.518743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfs2\" (UniqueName: \"kubernetes.io/projected/56e01e05-076f-41a9-8464-4176ceeb4d7d-kube-api-access-zwfs2\") pod \"s3-init-t26fm\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") " pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.619788 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.619747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e01e05-076f-41a9-8464-4176ceeb4d7d-model-cache\") pod \"s3-init-t26fm\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") " pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.620218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.619842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfs2\" (UniqueName: \"kubernetes.io/projected/56e01e05-076f-41a9-8464-4176ceeb4d7d-kube-api-access-zwfs2\") pod \"s3-init-t26fm\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") " pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.620218 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.620143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e01e05-076f-41a9-8464-4176ceeb4d7d-model-cache\") pod \"s3-init-t26fm\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") " pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.629186 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.629160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfs2\" (UniqueName: \"kubernetes.io/projected/56e01e05-076f-41a9-8464-4176ceeb4d7d-kube-api-access-zwfs2\") pod \"s3-init-t26fm\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") " pod="kserve/s3-init-t26fm"
Sep 05 02:27:37.797842 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:37.797748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t26fm"
Sep 05 02:27:38.086189 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:38.086165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t26fm"]
Sep 05 02:27:38.088826 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:27:38.088793    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e01e05_076f_41a9_8464_4176ceeb4d7d.slice/crio-3ebaa54221a55a8319137617ffecbd4f93c753c347ec2f9d8ecdbcc904940e76 WatchSource:0}: Error finding container 3ebaa54221a55a8319137617ffecbd4f93c753c347ec2f9d8ecdbcc904940e76: Status 404 returned error can't find the container with id 3ebaa54221a55a8319137617ffecbd4f93c753c347ec2f9d8ecdbcc904940e76
Sep 05 02:27:38.641217 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:38.641173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t26fm" event={"ID":"56e01e05-076f-41a9-8464-4176ceeb4d7d","Type":"ContainerStarted","Data":"3ebaa54221a55a8319137617ffecbd4f93c753c347ec2f9d8ecdbcc904940e76"}
Sep 05 02:27:41.218522 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:41.218498    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 05 02:27:41.657404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:41.657368    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t26fm" event={"ID":"56e01e05-076f-41a9-8464-4176ceeb4d7d","Type":"ContainerStarted","Data":"db8cc9e171c4d16d594584d56511f4eb9441ec682a26a46bf8eb9218b92d22b1"}
Sep 05 02:27:43.668413 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:43.668378    2582 generic.go:356] "Generic (PLEG): container finished" podID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerID="db8cc9e171c4d16d594584d56511f4eb9441ec682a26a46bf8eb9218b92d22b1" exitCode=0
Sep 05 02:27:43.668988 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:43.668465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t26fm" event={"ID":"56e01e05-076f-41a9-8464-4176ceeb4d7d","Type":"ContainerDied","Data":"db8cc9e171c4d16d594584d56511f4eb9441ec682a26a46bf8eb9218b92d22b1"}
Sep 05 02:27:48.698069 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:48.698037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t26fm" event={"ID":"56e01e05-076f-41a9-8464-4176ceeb4d7d","Type":"ContainerStarted","Data":"a82b3305dfe06cd4844ee22c8e923205900ec302c57fa49cf571c6a10e2a09bb"}
Sep 05 02:27:48.717315 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:27:48.717260    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-t26fm" podStartSLOduration=2.646436355 podStartE2EDuration="11.717242843s" totalImagesPullingTime="7.566278366s" totalInitContainerRuntime="1.504528122s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:48.714749494 +0000 UTC m=+609.136254833" watchObservedRunningTime="2026-09-05 02:27:48.717242843 +0000 UTC m=+609.138748181"
Sep 05 02:28:53.996612 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:53.996574    2582 generic.go:356] "Generic (PLEG): container finished" podID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerID="a82b3305dfe06cd4844ee22c8e923205900ec302c57fa49cf571c6a10e2a09bb" exitCode=0
Sep 05 02:28:53.997145 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:53.996625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t26fm" event={"ID":"56e01e05-076f-41a9-8464-4176ceeb4d7d","Type":"ContainerDied","Data":"a82b3305dfe06cd4844ee22c8e923205900ec302c57fa49cf571c6a10e2a09bb"}
Sep 05 02:28:55.134910 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.134881    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t26fm"
Sep 05 02:28:55.308221 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.308127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e01e05-076f-41a9-8464-4176ceeb4d7d-model-cache\") pod \"56e01e05-076f-41a9-8464-4176ceeb4d7d\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") "
Sep 05 02:28:55.308221 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.308176    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwfs2\" (UniqueName: \"kubernetes.io/projected/56e01e05-076f-41a9-8464-4176ceeb4d7d-kube-api-access-zwfs2\") pod \"56e01e05-076f-41a9-8464-4176ceeb4d7d\" (UID: \"56e01e05-076f-41a9-8464-4176ceeb4d7d\") "
Sep 05 02:28:55.310449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.310419    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e01e05-076f-41a9-8464-4176ceeb4d7d-kube-api-access-zwfs2" (OuterVolumeSpecName: "kube-api-access-zwfs2") pod "56e01e05-076f-41a9-8464-4176ceeb4d7d" (UID: "56e01e05-076f-41a9-8464-4176ceeb4d7d"). InnerVolumeSpecName "kube-api-access-zwfs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:28:55.327110 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.327067    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e01e05-076f-41a9-8464-4176ceeb4d7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "56e01e05-076f-41a9-8464-4176ceeb4d7d" (UID: "56e01e05-076f-41a9-8464-4176ceeb4d7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:28:55.408851 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.408816    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e01e05-076f-41a9-8464-4176ceeb4d7d-model-cache\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:28:55.408851 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:55.408844    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfs2\" (UniqueName: \"kubernetes.io/projected/56e01e05-076f-41a9-8464-4176ceeb4d7d-kube-api-access-zwfs2\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:28:56.007149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:56.007122    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t26fm"
Sep 05 02:28:56.007149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:56.007138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t26fm" event={"ID":"56e01e05-076f-41a9-8464-4176ceeb4d7d","Type":"ContainerDied","Data":"3ebaa54221a55a8319137617ffecbd4f93c753c347ec2f9d8ecdbcc904940e76"}
Sep 05 02:28:56.007363 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:28:56.007170    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ebaa54221a55a8319137617ffecbd4f93c753c347ec2f9d8ecdbcc904940e76"
Sep 05 02:29:38.436152 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.436073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"]
Sep 05 02:29:38.436586 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.436434    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerName="download-hf-model"
Sep 05 02:29:38.436586 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.436445    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerName="download-hf-model"
Sep 05 02:29:38.436586 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.436453    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerName="s3-init"
Sep 05 02:29:38.436586 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.436458    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerName="s3-init"
Sep 05 02:29:38.436586 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.436514    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e01e05-076f-41a9-8464-4176ceeb4d7d" containerName="s3-init"
Sep 05 02:29:38.439453 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.439433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.442019 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.441992    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a6d98-predictor-serving-cert\""
Sep 05 02:29:38.442019 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.442008    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:29:38.442229 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.442062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a6d98-kube-rbac-proxy-sar-config\""
Sep 05 02:29:38.442229 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.442063    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7f65m\""
Sep 05 02:29:38.442930 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.442915    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:29:38.449174 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.449150    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"]
Sep 05 02:29:38.581244 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.581207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-proxy-tls\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.581411 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.581286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.581411 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.581309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc94\" (UniqueName: \"kubernetes.io/projected/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-api-access-hbc94\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.682771 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.682736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-proxy-tls\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.682930 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.682784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.682930 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.682902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc94\" (UniqueName: \"kubernetes.io/projected/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-api-access-hbc94\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.683468 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.683446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.685290 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.685267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-proxy-tls\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.690832 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.690779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc94\" (UniqueName: \"kubernetes.io/projected/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-api-access-hbc94\") pod \"success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") " pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.712509 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.712481    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"]
Sep 05 02:29:38.716168 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.716149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.718652 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.718633    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:29:38.718749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.718657    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:29:38.718749 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.718698    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 05 02:29:38.718928 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.718909    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 05 02:29:38.726014 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.725995    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"]
Sep 05 02:29:38.750244 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.750216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:38.885140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.885103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0989e43-1e37-49ec-b85d-11df754904fc-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.885307 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.885147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.885307 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.885236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0989e43-1e37-49ec-b85d-11df754904fc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.885307 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.885280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.885421 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.885345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxpt\" (UniqueName: \"kubernetes.io/projected/b0989e43-1e37-49ec-b85d-11df754904fc-kube-api-access-9sxpt\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.888383 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.888361    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"]
Sep 05 02:29:38.892339 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.892321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:38.894888 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.894866    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 05 02:29:38.895009 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.894918    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 05 02:29:38.903301 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.903283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"]
Sep 05 02:29:38.985960 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.985870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0989e43-1e37-49ec-b85d-11df754904fc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.985960 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.985927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.986213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxpt\" (UniqueName: \"kubernetes.io/projected/b0989e43-1e37-49ec-b85d-11df754904fc-kube-api-access-9sxpt\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.986213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0989e43-1e37-49ec-b85d-11df754904fc-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.986213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:38.986213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d113cd26-11da-4e8d-8468-3619aae3a8d1-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:38.986421 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.986421 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927vp\" (UniqueName: \"kubernetes.io/projected/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-api-access-927vp\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:38.986421 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:38.986421 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d113cd26-11da-4e8d-8468-3619aae3a8d1-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:38.986549 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0989e43-1e37-49ec-b85d-11df754904fc-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.986684 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.986806 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.986789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:38.988457 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:38.988436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0989e43-1e37-49ec-b85d-11df754904fc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:39.004171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.004148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxpt\" (UniqueName: \"kubernetes.io/projected/b0989e43-1e37-49ec-b85d-11df754904fc-kube-api-access-9sxpt\") pod \"isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:39.027743 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.027711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:39.039559 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.039534    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"]
Sep 05 02:29:39.043876 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:29:39.043852    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfe7a8e_c944_468f_84b2_1a5f1ad15d40.slice/crio-a383a74fd371131e1968e12845cb6512f1a0fd669288f057dabc0717f93645f9 WatchSource:0}: Error finding container a383a74fd371131e1968e12845cb6512f1a0fd669288f057dabc0717f93645f9: Status 404 returned error can't find the container with id a383a74fd371131e1968e12845cb6512f1a0fd669288f057dabc0717f93645f9
Sep 05 02:29:39.046170 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.046154    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:29:39.087600 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.087567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.087600 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.087600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d113cd26-11da-4e8d-8468-3619aae3a8d1-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.088171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.087634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927vp\" (UniqueName: \"kubernetes.io/projected/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-api-access-927vp\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.088171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.087736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.088171 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.087791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d113cd26-11da-4e8d-8468-3619aae3a8d1-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.088350 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.088271    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.088475 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.088447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.088535 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.088517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d113cd26-11da-4e8d-8468-3619aae3a8d1-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.090546 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.090522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d113cd26-11da-4e8d-8468-3619aae3a8d1-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.100653 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.100600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-927vp\" (UniqueName: \"kubernetes.io/projected/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-api-access-927vp\") pod \"isvc-xgboost-graph-predictor-555fc69d56-bkcxs\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.204662 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.204633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:29:39.207025 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.206997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" event={"ID":"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40","Type":"ContainerStarted","Data":"a383a74fd371131e1968e12845cb6512f1a0fd669288f057dabc0717f93645f9"}
Sep 05 02:29:39.304083 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.301926    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"]
Sep 05 02:29:39.305307 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:29:39.305278    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0989e43_1e37_49ec_b85d_11df754904fc.slice/crio-88269f0d5da52b4afaceaae3caa4d590fcd1b771fe1c98778ccb944a42bb231d WatchSource:0}: Error finding container 88269f0d5da52b4afaceaae3caa4d590fcd1b771fe1c98778ccb944a42bb231d: Status 404 returned error can't find the container with id 88269f0d5da52b4afaceaae3caa4d590fcd1b771fe1c98778ccb944a42bb231d
Sep 05 02:29:39.479415 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:39.479362    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"]
Sep 05 02:29:39.481588 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:29:39.481559    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd113cd26_11da_4e8d_8468_3619aae3a8d1.slice/crio-6f88bc3a9386c9d79ef80c6ea69988bb1f3b386f5157f624ca97c50d4dac187a WatchSource:0}: Error finding container 6f88bc3a9386c9d79ef80c6ea69988bb1f3b386f5157f624ca97c50d4dac187a: Status 404 returned error can't find the container with id 6f88bc3a9386c9d79ef80c6ea69988bb1f3b386f5157f624ca97c50d4dac187a
Sep 05 02:29:40.226037 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:40.225990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerStarted","Data":"a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731"}
Sep 05 02:29:40.226225 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:40.226075    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerStarted","Data":"6f88bc3a9386c9d79ef80c6ea69988bb1f3b386f5157f624ca97c50d4dac187a"}
Sep 05 02:29:40.232708 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:40.232673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerStarted","Data":"de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b"}
Sep 05 02:29:40.232849 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:40.232745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerStarted","Data":"88269f0d5da52b4afaceaae3caa4d590fcd1b771fe1c98778ccb944a42bb231d"}
Sep 05 02:29:44.264107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:44.264079    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0989e43-1e37-49ec-b85d-11df754904fc" containerID="de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b" exitCode=0
Sep 05 02:29:44.264554 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:44.264151    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerDied","Data":"de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b"}
Sep 05 02:29:45.271568 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:45.271529    2582 generic.go:356] "Generic (PLEG): container finished" podID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerID="a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731" exitCode=0
Sep 05 02:29:45.272060 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:45.271610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerDied","Data":"a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731"}
Sep 05 02:29:51.316892 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:51.316761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" event={"ID":"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40","Type":"ContainerStarted","Data":"9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a"}
Sep 05 02:29:59.373115 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.373077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" event={"ID":"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40","Type":"ContainerStarted","Data":"823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d"}
Sep 05 02:29:59.373576 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.373151    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:59.373576 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.373328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:29:59.375120 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.375094    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:29:59.375754 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.375726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerStarted","Data":"ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2"}
Sep 05 02:29:59.375914 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.375785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerStarted","Data":"a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823"}
Sep 05 02:29:59.376023 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.376004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:29:59.392379 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.392330    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podStartSLOduration=2.010782831 podStartE2EDuration="21.392314677s" totalImagesPullingTime="19.381531846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:59.390721526 +0000 UTC m=+739.812226890" watchObservedRunningTime="2026-09-05 02:29:59.392314677 +0000 UTC m=+739.813820016"
Sep 05 02:29:59.411680 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:29:59.411616    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podStartSLOduration=2.838355859 podStartE2EDuration="21.411593953s" totalImagesPullingTime="14.18024182s" totalInitContainerRuntime="4.392996274s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:59.408993904 +0000 UTC m=+739.830499242" watchObservedRunningTime="2026-09-05 02:29:59.411593953 +0000 UTC m=+739.833099293"
Sep 05 02:30:00.382354 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:00.382298    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:00.382833 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:00.382405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:30:00.384318 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:00.384289    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:01.387662 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:01.387621    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:05.388175 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:05.388142    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:30:05.388898 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:05.388859    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:06.393471 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:06.393438    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:30:06.394112 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:06.394077    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:11.442738 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:11.442702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerStarted","Data":"c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726"}
Sep 05 02:30:11.443133 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:11.442767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerStarted","Data":"4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba"}
Sep 05 02:30:11.443133 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:11.443061    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:30:11.443232 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:11.443208    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:30:11.444581 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:11.444553    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:11.463615 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:11.463566    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podStartSLOduration=3.720889542 podStartE2EDuration="33.463550239s" totalImagesPullingTime="25.545469367s" totalInitContainerRuntime="4.19719133s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:11.461013479 +0000 UTC m=+751.882518817" watchObservedRunningTime="2026-09-05 02:30:11.463550239 +0000 UTC m=+751.885055577"
Sep 05 02:30:12.448182 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:12.448137    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:15.389408 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:15.389365    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:16.394697 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:16.394652    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:17.453775 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:17.453742    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:30:17.454281 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:17.454249    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:25.389477 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:25.389438    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:26.394640 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:26.394601    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:27.455029 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:27.454961    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:35.389019 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:35.388976    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 05 02:30:36.394131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:36.394093    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:37.455230 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:37.455186    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:45.390073 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:45.390040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:30:46.394033 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:46.393994    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:47.454680 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:47.454643    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:56.393982 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:56.393935    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:30:57.454473 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:30:57.454430    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:31:06.394021 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:06.393919    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:31:07.454669 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:07.454623    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:31:08.781884 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.781854    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"]
Sep 05 02:31:08.782361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.782196    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kube-rbac-proxy" containerID="cri-o://823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d" gracePeriod=30
Sep 05 02:31:08.782467 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.782423    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container" containerID="cri-o://9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a" gracePeriod=30
Sep 05 02:31:08.890277 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.890244    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"]
Sep 05 02:31:08.892926 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.892907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:08.895382 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.895357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3f635-predictor-serving-cert\""
Sep 05 02:31:08.895536 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.895514    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3f635-kube-rbac-proxy-sar-config\""
Sep 05 02:31:08.902128 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:08.902103    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"]
Sep 05 02:31:09.012201 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.012169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.012384 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.012220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mqp2\" (UniqueName: \"kubernetes.io/projected/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-api-access-7mqp2\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.012384 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.012250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.112837 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.112800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.113042 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.112852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mqp2\" (UniqueName: \"kubernetes.io/projected/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-api-access-7mqp2\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.113042 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.112878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.113042 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:31:09.113030    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3f635-predictor-serving-cert: secret "success-200-isvc-3f635-predictor-serving-cert" not found
Sep 05 02:31:09.113234 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:31:09.113098    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls podName:d1a861f7-aa26-4ce0-9e61-40badb532623 nodeName:}" failed. No retries permitted until 2026-09-05 02:31:09.613083188 +0000 UTC m=+810.034588504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls") pod "success-200-isvc-3f635-predictor-697669ffb4-bjr2d" (UID: "d1a861f7-aa26-4ce0-9e61-40badb532623") : secret "success-200-isvc-3f635-predictor-serving-cert" not found
Sep 05 02:31:09.113537 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.113506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.122062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.122037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mqp2\" (UniqueName: \"kubernetes.io/projected/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-api-access-7mqp2\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.618499 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.618461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.620796 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.620774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls\") pod \"success-200-isvc-3f635-predictor-697669ffb4-bjr2d\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") " pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:09.729796 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.729762    2582 generic.go:356] "Generic (PLEG): container finished" podID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerID="823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d" exitCode=2
Sep 05 02:31:09.729957 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.729813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" event={"ID":"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40","Type":"ContainerDied","Data":"823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d"}
Sep 05 02:31:09.804623 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:09.804586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:10.077752 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.077555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"]
Sep 05 02:31:10.080413 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:31:10.080388    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a861f7_aa26_4ce0_9e61_40badb532623.slice/crio-31dbb215be172b28342ac8f4d87afb57a58af774750b5e11d8c668da825bdd96 WatchSource:0}: Error finding container 31dbb215be172b28342ac8f4d87afb57a58af774750b5e11d8c668da825bdd96: Status 404 returned error can't find the container with id 31dbb215be172b28342ac8f4d87afb57a58af774750b5e11d8c668da825bdd96
Sep 05 02:31:10.383029 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.382933    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 05 02:31:10.736699 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.736613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" event={"ID":"d1a861f7-aa26-4ce0-9e61-40badb532623","Type":"ContainerStarted","Data":"353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61"}
Sep 05 02:31:10.736699 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.736683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" event={"ID":"d1a861f7-aa26-4ce0-9e61-40badb532623","Type":"ContainerStarted","Data":"db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138"}
Sep 05 02:31:10.736886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.736712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" event={"ID":"d1a861f7-aa26-4ce0-9e61-40badb532623","Type":"ContainerStarted","Data":"31dbb215be172b28342ac8f4d87afb57a58af774750b5e11d8c668da825bdd96"}
Sep 05 02:31:10.736886 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.736861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:10.757491 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:10.757424    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podStartSLOduration=2.757401671 podStartE2EDuration="2.757401671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:10.755800094 +0000 UTC m=+811.177305435" watchObservedRunningTime="2026-09-05 02:31:10.757401671 +0000 UTC m=+811.178907011"
Sep 05 02:31:11.741341 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:11.741307    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:11.742757 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:11.742725    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:12.441561 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.441527    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:31:12.545086 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.544950    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbc94\" (UniqueName: \"kubernetes.io/projected/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-api-access-hbc94\") pod \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") "
Sep 05 02:31:12.545086 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.545071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-rbac-proxy-sar-config\") pod \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") "
Sep 05 02:31:12.545327 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.545125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-proxy-tls\") pod \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\" (UID: \"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40\") "
Sep 05 02:31:12.545438 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.545413    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" (UID: "ccfe7a8e-c944-468f-84b2-1a5f1ad15d40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:31:12.547294 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.547259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" (UID: "ccfe7a8e-c944-468f-84b2-1a5f1ad15d40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:31:12.547394 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.547268    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-api-access-hbc94" (OuterVolumeSpecName: "kube-api-access-hbc94") pod "ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" (UID: "ccfe7a8e-c944-468f-84b2-1a5f1ad15d40"). InnerVolumeSpecName "kube-api-access-hbc94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:31:12.646149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.646114    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:31:12.646149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.646145    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbc94\" (UniqueName: \"kubernetes.io/projected/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-api-access-hbc94\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:31:12.646149 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.646156    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:31:12.748432 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.748399    2582 generic.go:356] "Generic (PLEG): container finished" podID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerID="9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a" exitCode=0
Sep 05 02:31:12.748919 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.748459    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"
Sep 05 02:31:12.748919 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.748490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" event={"ID":"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40","Type":"ContainerDied","Data":"9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a"}
Sep 05 02:31:12.748919 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.748544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9" event={"ID":"ccfe7a8e-c944-468f-84b2-1a5f1ad15d40","Type":"ContainerDied","Data":"a383a74fd371131e1968e12845cb6512f1a0fd669288f057dabc0717f93645f9"}
Sep 05 02:31:12.748919 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.748572    2582 scope.go:117] "RemoveContainer" containerID="823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d"
Sep 05 02:31:12.749333 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.749255    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:12.758526 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.758500    2582 scope.go:117] "RemoveContainer" containerID="9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a"
Sep 05 02:31:12.768533 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.768499    2582 scope.go:117] "RemoveContainer" containerID="823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d"
Sep 05 02:31:12.769041 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:31:12.769017    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d\": container with ID starting with 823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d not found: ID does not exist" containerID="823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d"
Sep 05 02:31:12.769114 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.769050    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d"} err="failed to get container status \"823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d\": rpc error: code = NotFound desc = could not find container \"823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d\": container with ID starting with 823f18284adfdb001a1dae2d04339992018280e1c9715d195a24a2a916a35b1d not found: ID does not exist"
Sep 05 02:31:12.769114 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.769068    2582 scope.go:117] "RemoveContainer" containerID="9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a"
Sep 05 02:31:12.769361 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:31:12.769345    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a\": container with ID starting with 9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a not found: ID does not exist" containerID="9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a"
Sep 05 02:31:12.769406 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.769365    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a"} err="failed to get container status \"9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a\": rpc error: code = NotFound desc = could not find container \"9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a\": container with ID starting with 9707e13ce48f239289cca37e725a020e24cc92ca280b4be8b11acbb6f10b3c9a not found: ID does not exist"
Sep 05 02:31:12.773564 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.773540    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"]
Sep 05 02:31:12.776472 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:12.776447    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a6d98-predictor-56646fb6cd-8cfd9"]
Sep 05 02:31:14.110375 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:14.110341    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" path="/var/lib/kubelet/pods/ccfe7a8e-c944-468f-84b2-1a5f1ad15d40/volumes"
Sep 05 02:31:16.394276 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:16.394234    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 05 02:31:17.456162 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:17.456133    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:31:17.753847 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:17.753768    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:17.754294 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:17.754257    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:26.395164 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:26.395131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:31:27.754412 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:27.754371    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:37.754268 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:37.754218    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:47.755053 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:47.755005    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:57.755049 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:57.755013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:31:58.703084 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.703054    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"]
Sep 05 02:31:58.703389 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.703358    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container" containerID="cri-o://a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823" gracePeriod=30
Sep 05 02:31:58.703509 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.703361    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kube-rbac-proxy" containerID="cri-o://ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2" gracePeriod=30
Sep 05 02:31:58.737199 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.737170    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"]
Sep 05 02:31:58.737435 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.737400    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container" containerID="cri-o://4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba" gracePeriod=30
Sep 05 02:31:58.737558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.737435    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kube-rbac-proxy" containerID="cri-o://c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726" gracePeriod=30
Sep 05 02:31:58.817145 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817116    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"]
Sep 05 02:31:58.817547 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817532    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container"
Sep 05 02:31:58.817602 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817549    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container"
Sep 05 02:31:58.817602 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817563    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kube-rbac-proxy"
Sep 05 02:31:58.817602 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817570    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kube-rbac-proxy"
Sep 05 02:31:58.817705 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817635    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kserve-container"
Sep 05 02:31:58.817705 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.817644    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfe7a8e-c944-468f-84b2-1a5f1ad15d40" containerName="kube-rbac-proxy"
Sep 05 02:31:58.819547 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.819531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:58.822197 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.822153    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9f825-predictor-serving-cert\""
Sep 05 02:31:58.822352 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.822329    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9f825-kube-rbac-proxy-sar-config\""
Sep 05 02:31:58.832514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.832488    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"]
Sep 05 02:31:58.970096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.970008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9pk\" (UniqueName: \"kubernetes.io/projected/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-api-access-sh9pk\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:58.970096 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.970083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:58.970334 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.970110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3923ab1-65a6-4eca-80a4-95246afae40e-proxy-tls\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:58.971885 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.971859    2582 generic.go:356] "Generic (PLEG): container finished" podID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerID="c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726" exitCode=2
Sep 05 02:31:58.972040 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.971930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerDied","Data":"c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726"}
Sep 05 02:31:58.974109 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.974088    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0989e43-1e37-49ec-b85d-11df754904fc" containerID="ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2" exitCode=2
Sep 05 02:31:58.974202 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:58.974164    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerDied","Data":"ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2"}
Sep 05 02:31:59.070955 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.070920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.071161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.070959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3923ab1-65a6-4eca-80a4-95246afae40e-proxy-tls\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.071161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.071061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9pk\" (UniqueName: \"kubernetes.io/projected/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-api-access-sh9pk\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.071575 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.071550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.073389 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.073372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3923ab1-65a6-4eca-80a4-95246afae40e-proxy-tls\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.080005 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.079981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9pk\" (UniqueName: \"kubernetes.io/projected/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-api-access-sh9pk\") pod \"success-200-isvc-9f825-predictor-6cfc7876b6-flxxb\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") " pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.129623 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.129588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.403434 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.403255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"]
Sep 05 02:31:59.406098 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:31:59.406070    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3923ab1_65a6_4eca_80a4_95246afae40e.slice/crio-b59b9566378830f96bcd2a237a56b4defcce4065c58c8cef31459304b13e8d1b WatchSource:0}: Error finding container b59b9566378830f96bcd2a237a56b4defcce4065c58c8cef31459304b13e8d1b: Status 404 returned error can't find the container with id b59b9566378830f96bcd2a237a56b4defcce4065c58c8cef31459304b13e8d1b
Sep 05 02:31:59.981061 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.981025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" event={"ID":"a3923ab1-65a6-4eca-80a4-95246afae40e","Type":"ContainerStarted","Data":"b820a8bc85acc4b0d2db38a988295f20aacf5428ca3c04ed8858d3b35de7d6b6"}
Sep 05 02:31:59.981431 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.981093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" event={"ID":"a3923ab1-65a6-4eca-80a4-95246afae40e","Type":"ContainerStarted","Data":"0b7eeb587bdd4b413e8b0440d095f6423ef09cdd01a85407ea15a3a4c33cd8b6"}
Sep 05 02:31:59.981431 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.981128    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:31:59.981431 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:31:59.981156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" event={"ID":"a3923ab1-65a6-4eca-80a4-95246afae40e","Type":"ContainerStarted","Data":"b59b9566378830f96bcd2a237a56b4defcce4065c58c8cef31459304b13e8d1b"}
Sep 05 02:32:00.000925 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:00.000877    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podStartSLOduration=2.000849604 podStartE2EDuration="2.000849604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:59.999307451 +0000 UTC m=+860.420812785" watchObservedRunningTime="2026-09-05 02:32:00.000849604 +0000 UTC m=+860.422355013"
Sep 05 02:32:00.985933 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:00.985903    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:32:00.987430 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:00.987395    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:01.388650 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:01.388609    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 05 02:32:01.990907 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:01.990861    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:02.448567 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.448531    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:32:02.588379 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.588355    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:32:02.702776 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.702690    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-cabundle-cert\") pod \"d113cd26-11da-4e8d-8468-3619aae3a8d1\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") "
Sep 05 02:32:02.702776 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.702752    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d113cd26-11da-4e8d-8468-3619aae3a8d1-proxy-tls\") pod \"d113cd26-11da-4e8d-8468-3619aae3a8d1\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") "
Sep 05 02:32:02.702776 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.702777    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d113cd26-11da-4e8d-8468-3619aae3a8d1-kserve-provision-location\") pod \"d113cd26-11da-4e8d-8468-3619aae3a8d1\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") "
Sep 05 02:32:02.703068 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.702821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-927vp\" (UniqueName: \"kubernetes.io/projected/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-api-access-927vp\") pod \"d113cd26-11da-4e8d-8468-3619aae3a8d1\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") "
Sep 05 02:32:02.703068 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.702848    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-rbac-proxy-sar-config\") pod \"d113cd26-11da-4e8d-8468-3619aae3a8d1\" (UID: \"d113cd26-11da-4e8d-8468-3619aae3a8d1\") "
Sep 05 02:32:02.703195 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.703167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d113cd26-11da-4e8d-8468-3619aae3a8d1" (UID: "d113cd26-11da-4e8d-8468-3619aae3a8d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:02.703252 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.703167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d113cd26-11da-4e8d-8468-3619aae3a8d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d113cd26-11da-4e8d-8468-3619aae3a8d1" (UID: "d113cd26-11da-4e8d-8468-3619aae3a8d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:02.703295 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.703245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d113cd26-11da-4e8d-8468-3619aae3a8d1" (UID: "d113cd26-11da-4e8d-8468-3619aae3a8d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:02.704994 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.704954    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d113cd26-11da-4e8d-8468-3619aae3a8d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d113cd26-11da-4e8d-8468-3619aae3a8d1" (UID: "d113cd26-11da-4e8d-8468-3619aae3a8d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:02.705076 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.705049    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-api-access-927vp" (OuterVolumeSpecName: "kube-api-access-927vp") pod "d113cd26-11da-4e8d-8468-3619aae3a8d1" (UID: "d113cd26-11da-4e8d-8468-3619aae3a8d1"). InnerVolumeSpecName "kube-api-access-927vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:02.804385 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.804329    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927vp\" (UniqueName: \"kubernetes.io/projected/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-api-access-927vp\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:02.804385 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.804374    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:02.804385 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.804386    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d113cd26-11da-4e8d-8468-3619aae3a8d1-cabundle-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:02.804385 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.804396    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d113cd26-11da-4e8d-8468-3619aae3a8d1-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:02.804385 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.804405    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d113cd26-11da-4e8d-8468-3619aae3a8d1-kserve-provision-location\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:02.997867 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.997778    2582 generic.go:356] "Generic (PLEG): container finished" podID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerID="4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba" exitCode=0
Sep 05 02:32:02.997867 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.997837    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"
Sep 05 02:32:02.997867 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.997854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerDied","Data":"4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba"}
Sep 05 02:32:02.998366 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.997936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs" event={"ID":"d113cd26-11da-4e8d-8468-3619aae3a8d1","Type":"ContainerDied","Data":"6f88bc3a9386c9d79ef80c6ea69988bb1f3b386f5157f624ca97c50d4dac187a"}
Sep 05 02:32:02.998366 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:02.997994    2582 scope.go:117] "RemoveContainer" containerID="c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726"
Sep 05 02:32:03.007107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.007087    2582 scope.go:117] "RemoveContainer" containerID="4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba"
Sep 05 02:32:03.015062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.015044    2582 scope.go:117] "RemoveContainer" containerID="a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731"
Sep 05 02:32:03.020830 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.020806    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"]
Sep 05 02:32:03.023877 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.023856    2582 scope.go:117] "RemoveContainer" containerID="c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726"
Sep 05 02:32:03.024509 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:32:03.024364    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726\": container with ID starting with c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726 not found: ID does not exist" containerID="c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726"
Sep 05 02:32:03.024509 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.024399    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726"} err="failed to get container status \"c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726\": rpc error: code = NotFound desc = could not find container \"c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726\": container with ID starting with c07362f2faa6651ad1cec76b6bbbe51c5416f5a780ee0f9624c70d1671e39726 not found: ID does not exist"
Sep 05 02:32:03.024509 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.024423    2582 scope.go:117] "RemoveContainer" containerID="4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba"
Sep 05 02:32:03.024950 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:32:03.024821    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba\": container with ID starting with 4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba not found: ID does not exist" containerID="4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba"
Sep 05 02:32:03.024950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.024856    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba"} err="failed to get container status \"4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba\": rpc error: code = NotFound desc = could not find container \"4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba\": container with ID starting with 4a07e59fa3417041ab4322b88500ddd2d014ce61b2254a5cebe19b8d0d80a1ba not found: ID does not exist"
Sep 05 02:32:03.024950 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.024877    2582 scope.go:117] "RemoveContainer" containerID="a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731"
Sep 05 02:32:03.025386 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:32:03.025359    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731\": container with ID starting with a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731 not found: ID does not exist" containerID="a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731"
Sep 05 02:32:03.025470 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.025390    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731"} err="failed to get container status \"a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731\": rpc error: code = NotFound desc = could not find container \"a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731\": container with ID starting with a0363ff4f249b08817af113df4c8382ddf5cbd1f9ec97051a73322a0af93d731 not found: ID does not exist"
Sep 05 02:32:03.026007 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.025984    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-bkcxs"]
Sep 05 02:32:03.643932 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.643908    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:32:03.812970 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.812943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0989e43-1e37-49ec-b85d-11df754904fc-proxy-tls\") pod \"b0989e43-1e37-49ec-b85d-11df754904fc\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") "
Sep 05 02:32:03.813153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-cabundle-cert\") pod \"b0989e43-1e37-49ec-b85d-11df754904fc\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") "
Sep 05 02:32:03.813153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813057    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0989e43-1e37-49ec-b85d-11df754904fc-kserve-provision-location\") pod \"b0989e43-1e37-49ec-b85d-11df754904fc\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") "
Sep 05 02:32:03.813153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-kube-rbac-proxy-sar-config\") pod \"b0989e43-1e37-49ec-b85d-11df754904fc\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") "
Sep 05 02:32:03.813153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sxpt\" (UniqueName: \"kubernetes.io/projected/b0989e43-1e37-49ec-b85d-11df754904fc-kube-api-access-9sxpt\") pod \"b0989e43-1e37-49ec-b85d-11df754904fc\" (UID: \"b0989e43-1e37-49ec-b85d-11df754904fc\") "
Sep 05 02:32:03.813404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0989e43-1e37-49ec-b85d-11df754904fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0989e43-1e37-49ec-b85d-11df754904fc" (UID: "b0989e43-1e37-49ec-b85d-11df754904fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:03.813404 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0989e43-1e37-49ec-b85d-11df754904fc" (UID: "b0989e43-1e37-49ec-b85d-11df754904fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:03.813489 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0989e43-1e37-49ec-b85d-11df754904fc" (UID: "b0989e43-1e37-49ec-b85d-11df754904fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:03.813544 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813530    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-cabundle-cert\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:03.813581 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813547    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0989e43-1e37-49ec-b85d-11df754904fc-kserve-provision-location\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:03.813581 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.813557    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0989e43-1e37-49ec-b85d-11df754904fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:03.815121 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.815102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0989e43-1e37-49ec-b85d-11df754904fc-kube-api-access-9sxpt" (OuterVolumeSpecName: "kube-api-access-9sxpt") pod "b0989e43-1e37-49ec-b85d-11df754904fc" (UID: "b0989e43-1e37-49ec-b85d-11df754904fc"). InnerVolumeSpecName "kube-api-access-9sxpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:03.815270 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.815251    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0989e43-1e37-49ec-b85d-11df754904fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0989e43-1e37-49ec-b85d-11df754904fc" (UID: "b0989e43-1e37-49ec-b85d-11df754904fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:03.914952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.914917    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0989e43-1e37-49ec-b85d-11df754904fc-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:03.914952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:03.914947    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxpt\" (UniqueName: \"kubernetes.io/projected/b0989e43-1e37-49ec-b85d-11df754904fc-kube-api-access-9sxpt\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:32:04.004821 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.004786    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0989e43-1e37-49ec-b85d-11df754904fc" containerID="a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823" exitCode=0
Sep 05 02:32:04.005272 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.004843    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"
Sep 05 02:32:04.005272 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.004864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerDied","Data":"a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823"}
Sep 05 02:32:04.005272 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.004922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw" event={"ID":"b0989e43-1e37-49ec-b85d-11df754904fc","Type":"ContainerDied","Data":"88269f0d5da52b4afaceaae3caa4d590fcd1b771fe1c98778ccb944a42bb231d"}
Sep 05 02:32:04.005272 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.004949    2582 scope.go:117] "RemoveContainer" containerID="ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2"
Sep 05 02:32:04.013596 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.013577    2582 scope.go:117] "RemoveContainer" containerID="a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823"
Sep 05 02:32:04.021643 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.021623    2582 scope.go:117] "RemoveContainer" containerID="de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b"
Sep 05 02:32:04.027680 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.027657    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"]
Sep 05 02:32:04.029756 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.029729    2582 scope.go:117] "RemoveContainer" containerID="ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2"
Sep 05 02:32:04.030099 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:32:04.030055    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2\": container with ID starting with ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2 not found: ID does not exist" containerID="ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2"
Sep 05 02:32:04.030170 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.030111    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2"} err="failed to get container status \"ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2\": rpc error: code = NotFound desc = could not find container \"ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2\": container with ID starting with ef206acba4002f692c804e3f9a4d1be974facf4b80fd2b352e32a89a9614d7b2 not found: ID does not exist"
Sep 05 02:32:04.030170 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.030158    2582 scope.go:117] "RemoveContainer" containerID="a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823"
Sep 05 02:32:04.030637 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:32:04.030612    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823\": container with ID starting with a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823 not found: ID does not exist" containerID="a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823"
Sep 05 02:32:04.030747 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.030680    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823"} err="failed to get container status \"a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823\": rpc error: code = NotFound desc = could not find container \"a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823\": container with ID starting with a8af67b73ff6faee0e5bcab17a9335d7a3f366a677909b8ae734a0aa23728823 not found: ID does not exist"
Sep 05 02:32:04.030747 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.030708    2582 scope.go:117] "RemoveContainer" containerID="de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b"
Sep 05 02:32:04.031108 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:32:04.031070    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b\": container with ID starting with de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b not found: ID does not exist" containerID="de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b"
Sep 05 02:32:04.031189 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.031117    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b"} err="failed to get container status \"de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b\": rpc error: code = NotFound desc = could not find container \"de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b\": container with ID starting with de299f691f34151d8ad182dae08f808d163f09dc414f27e3641e4e202486700b not found: ID does not exist"
Sep 05 02:32:04.032644 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.032626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5579fcbfd8-2dhtw"]
Sep 05 02:32:04.109271 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.109194    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" path="/var/lib/kubelet/pods/b0989e43-1e37-49ec-b85d-11df754904fc/volumes"
Sep 05 02:32:04.109702 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:04.109688    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" path="/var/lib/kubelet/pods/d113cd26-11da-4e8d-8468-3619aae3a8d1/volumes"
Sep 05 02:32:06.995729 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:06.995693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:32:06.996229 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:06.996173    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:16.996361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:16.996315    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:26.997107 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:26.997062    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:36.996516 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:36.996431    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:46.997544 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:32:46.997502    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:40:23.690733 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.690701    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"]
Sep 05 02:40:23.691250 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.690977    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container" containerID="cri-o://db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138" gracePeriod=30
Sep 05 02:40:23.691250 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.691004    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kube-rbac-proxy" containerID="cri-o://353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61" gracePeriod=30
Sep 05 02:40:23.767191 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767157    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"]
Sep 05 02:40:23.767532 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767520    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container"
Sep 05 02:40:23.767582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767534    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container"
Sep 05 02:40:23.767582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767544    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kube-rbac-proxy"
Sep 05 02:40:23.767582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767550    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kube-rbac-proxy"
Sep 05 02:40:23.767582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="storage-initializer"
Sep 05 02:40:23.767582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767569    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="storage-initializer"
Sep 05 02:40:23.767582 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767581    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="storage-initializer"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767587    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="storage-initializer"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767595    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767601    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kube-rbac-proxy"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767611    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kube-rbac-proxy"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767666    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kube-rbac-proxy"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767673    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0989e43-1e37-49ec-b85d-11df754904fc" containerName="kserve-container"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767678    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kube-rbac-proxy"
Sep 05 02:40:23.767800 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.767685    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d113cd26-11da-4e8d-8468-3619aae3a8d1" containerName="kserve-container"
Sep 05 02:40:23.770907 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.770889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.773911 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.773889    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-12d88-predictor-serving-cert\""
Sep 05 02:40:23.774244 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.774214    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-12d88-kube-rbac-proxy-sar-config\""
Sep 05 02:40:23.783502 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.783477    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"]
Sep 05 02:40:23.835282 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.835249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtmk\" (UniqueName: \"kubernetes.io/projected/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-api-access-pbtmk\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.835439 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.835289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.835439 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.835309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.936475 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.936441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtmk\" (UniqueName: \"kubernetes.io/projected/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-api-access-pbtmk\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.936652 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.936487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.936652 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.936513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.936652 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:40:23.936592    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-12d88-predictor-serving-cert: secret "success-200-isvc-12d88-predictor-serving-cert" not found
Sep 05 02:40:23.936823 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:40:23.936665    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls podName:e187ed0e-36e7-49d0-9bd0-141378377e57 nodeName:}" failed. No retries permitted until 2026-09-05 02:40:24.436643681 +0000 UTC m=+1364.858148998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls") pod "success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" (UID: "e187ed0e-36e7-49d0-9bd0-141378377e57") : secret "success-200-isvc-12d88-predictor-serving-cert" not found
Sep 05 02:40:23.937176 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.937158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:23.945225 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:23.945171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtmk\" (UniqueName: \"kubernetes.io/projected/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-api-access-pbtmk\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:24.373457 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:24.373423    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerID="353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61" exitCode=2
Sep 05 02:40:24.373679 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:24.373492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" event={"ID":"d1a861f7-aa26-4ce0-9e61-40badb532623","Type":"ContainerDied","Data":"353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61"}
Sep 05 02:40:24.441589 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:24.441547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:24.443952 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:24.443927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls\") pod \"success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") " pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:24.684472 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:24.684373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:24.954611 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:24.954532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"]
Sep 05 02:40:24.958427 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:40:24.958401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode187ed0e_36e7_49d0_9bd0_141378377e57.slice/crio-fb3092fb29395cbb29fdbeb4146b731544acf6fe35bdd4012042c636747ebc7d WatchSource:0}: Error finding container fb3092fb29395cbb29fdbeb4146b731544acf6fe35bdd4012042c636747ebc7d: Status 404 returned error can't find the container with id fb3092fb29395cbb29fdbeb4146b731544acf6fe35bdd4012042c636747ebc7d
Sep 05 02:40:25.380981 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:25.380930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" event={"ID":"e187ed0e-36e7-49d0-9bd0-141378377e57","Type":"ContainerStarted","Data":"4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0"}
Sep 05 02:40:25.381158 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:25.381015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" event={"ID":"e187ed0e-36e7-49d0-9bd0-141378377e57","Type":"ContainerStarted","Data":"dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae"}
Sep 05 02:40:25.381158 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:25.381045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" event={"ID":"e187ed0e-36e7-49d0-9bd0-141378377e57","Type":"ContainerStarted","Data":"fb3092fb29395cbb29fdbeb4146b731544acf6fe35bdd4012042c636747ebc7d"}
Sep 05 02:40:25.381238 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:25.381185    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:25.400734 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:25.400684    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podStartSLOduration=2.400668429 podStartE2EDuration="2.400668429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:25.399166462 +0000 UTC m=+1365.820671839" watchObservedRunningTime="2026-09-05 02:40:25.400668429 +0000 UTC m=+1365.822173767"
Sep 05 02:40:26.385581 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:26.385546    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:26.386906 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:26.386879    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:27.033759 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.033735    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:40:27.066202 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.066172    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mqp2\" (UniqueName: \"kubernetes.io/projected/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-api-access-7mqp2\") pod \"d1a861f7-aa26-4ce0-9e61-40badb532623\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") "
Sep 05 02:40:27.066361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.066215    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls\") pod \"d1a861f7-aa26-4ce0-9e61-40badb532623\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") "
Sep 05 02:40:27.066361 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.066242    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-rbac-proxy-sar-config\") pod \"d1a861f7-aa26-4ce0-9e61-40badb532623\" (UID: \"d1a861f7-aa26-4ce0-9e61-40badb532623\") "
Sep 05 02:40:27.066678 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.066649    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1a861f7-aa26-4ce0-9e61-40badb532623" (UID: "d1a861f7-aa26-4ce0-9e61-40badb532623"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:27.068117 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.068091    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-api-access-7mqp2" (OuterVolumeSpecName: "kube-api-access-7mqp2") pod "d1a861f7-aa26-4ce0-9e61-40badb532623" (UID: "d1a861f7-aa26-4ce0-9e61-40badb532623"). InnerVolumeSpecName "kube-api-access-7mqp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:27.068282 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.068262    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1a861f7-aa26-4ce0-9e61-40badb532623" (UID: "d1a861f7-aa26-4ce0-9e61-40badb532623"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:27.167570 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.167467    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mqp2\" (UniqueName: \"kubernetes.io/projected/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-api-access-7mqp2\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:40:27.167570 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.167509    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1a861f7-aa26-4ce0-9e61-40badb532623-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:40:27.167570 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.167525    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1a861f7-aa26-4ce0-9e61-40badb532623-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:40:27.392011 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.391949    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerID="db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138" exitCode=0
Sep 05 02:40:27.392011 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.392004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" event={"ID":"d1a861f7-aa26-4ce0-9e61-40badb532623","Type":"ContainerDied","Data":"db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138"}
Sep 05 02:40:27.392484 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.392025    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"
Sep 05 02:40:27.392484 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.392063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d" event={"ID":"d1a861f7-aa26-4ce0-9e61-40badb532623","Type":"ContainerDied","Data":"31dbb215be172b28342ac8f4d87afb57a58af774750b5e11d8c668da825bdd96"}
Sep 05 02:40:27.392484 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.392090    2582 scope.go:117] "RemoveContainer" containerID="353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61"
Sep 05 02:40:27.392647 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.392556    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:27.401399 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.401378    2582 scope.go:117] "RemoveContainer" containerID="db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138"
Sep 05 02:40:27.411413 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.411388    2582 scope.go:117] "RemoveContainer" containerID="353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61"
Sep 05 02:40:27.412208 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:40:27.412183    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61\": container with ID starting with 353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61 not found: ID does not exist" containerID="353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61"
Sep 05 02:40:27.412301 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.412216    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61"} err="failed to get container status \"353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61\": rpc error: code = NotFound desc = could not find container \"353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61\": container with ID starting with 353a797cf4997cfdd7ef4e67a6c83c6e326c1e7f719a5ff9f914021e7f4bbe61 not found: ID does not exist"
Sep 05 02:40:27.412301 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.412233    2582 scope.go:117] "RemoveContainer" containerID="db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138"
Sep 05 02:40:27.412562 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:40:27.412542    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138\": container with ID starting with db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138 not found: ID does not exist" containerID="db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138"
Sep 05 02:40:27.412638 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.412573    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138"} err="failed to get container status \"db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138\": rpc error: code = NotFound desc = could not find container \"db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138\": container with ID starting with db3899e06145a5b6edcc67e106bac345f09132b9ae5bb6937497ce508cc14138 not found: ID does not exist"
Sep 05 02:40:27.413777 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.413758    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"]
Sep 05 02:40:27.416769 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:27.416748    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3f635-predictor-697669ffb4-bjr2d"]
Sep 05 02:40:28.110247 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:28.110213    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" path="/var/lib/kubelet/pods/d1a861f7-aa26-4ce0-9e61-40badb532623/volumes"
Sep 05 02:40:32.397181 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:32.397153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:40:32.397610 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:32.397584    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:42.398044 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:42.397999    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:40:52.398140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:40:52.398094    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:41:02.397919 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:02.397880    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:41:12.399131 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:12.399100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:41:13.570477 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.570446    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"]
Sep 05 02:41:13.570881 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.570684    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container" containerID="cri-o://0b7eeb587bdd4b413e8b0440d095f6423ef09cdd01a85407ea15a3a4c33cd8b6" gracePeriod=30
Sep 05 02:41:13.570881 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.570737    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kube-rbac-proxy" containerID="cri-o://b820a8bc85acc4b0d2db38a988295f20aacf5428ca3c04ed8858d3b35de7d6b6" gracePeriod=30
Sep 05 02:41:13.694043 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694007    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"]
Sep 05 02:41:13.694416 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694403    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container"
Sep 05 02:41:13.694473 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694417    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container"
Sep 05 02:41:13.694473 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694436    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kube-rbac-proxy"
Sep 05 02:41:13.694473 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694442    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kube-rbac-proxy"
Sep 05 02:41:13.694574 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kserve-container"
Sep 05 02:41:13.694574 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.694513    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a861f7-aa26-4ce0-9e61-40badb532623" containerName="kube-rbac-proxy"
Sep 05 02:41:13.697554 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.697530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.700100 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.700077    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-80c39-predictor-serving-cert\""
Sep 05 02:41:13.700246 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.700085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-80c39-kube-rbac-proxy-sar-config\""
Sep 05 02:41:13.707712 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.707686    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"]
Sep 05 02:41:13.781933 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.781887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95abbc15-baab-4701-8410-70a2d6621a5c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.782140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.782016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95abbc15-baab-4701-8410-70a2d6621a5c-proxy-tls\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.782140 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.782052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzt6w\" (UniqueName: \"kubernetes.io/projected/95abbc15-baab-4701-8410-70a2d6621a5c-kube-api-access-zzt6w\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.883408 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.883315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95abbc15-baab-4701-8410-70a2d6621a5c-proxy-tls\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.883408 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.883357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzt6w\" (UniqueName: \"kubernetes.io/projected/95abbc15-baab-4701-8410-70a2d6621a5c-kube-api-access-zzt6w\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.883408 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.883403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95abbc15-baab-4701-8410-70a2d6621a5c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.884134 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.884113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95abbc15-baab-4701-8410-70a2d6621a5c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.885912 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.885887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95abbc15-baab-4701-8410-70a2d6621a5c-proxy-tls\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:13.893251 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:13.893230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzt6w\" (UniqueName: \"kubernetes.io/projected/95abbc15-baab-4701-8410-70a2d6621a5c-kube-api-access-zzt6w\") pod \"success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") " pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:14.009205 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.009165    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:14.285013 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.284987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"]
Sep 05 02:41:14.286633 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:41:14.286605    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95abbc15_baab_4701_8410_70a2d6621a5c.slice/crio-97c7b76893865c85f30f9d9be3d8b7538d4b7ff743d482fcf56ca12552906c5f WatchSource:0}: Error finding container 97c7b76893865c85f30f9d9be3d8b7538d4b7ff743d482fcf56ca12552906c5f: Status 404 returned error can't find the container with id 97c7b76893865c85f30f9d9be3d8b7538d4b7ff743d482fcf56ca12552906c5f
Sep 05 02:41:14.622032 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.621995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" event={"ID":"95abbc15-baab-4701-8410-70a2d6621a5c","Type":"ContainerStarted","Data":"9d5ba700ca9581b0b4962fe02a4d7af7f3f48aedc72cc29f19accb91997acaca"}
Sep 05 02:41:14.622592 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.622064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" event={"ID":"95abbc15-baab-4701-8410-70a2d6621a5c","Type":"ContainerStarted","Data":"584e47f6ae9254bd32ab03e30f386928f9d710775d122a0709060ba044c538fa"}
Sep 05 02:41:14.622592 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.622106    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:14.622592 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.622127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" event={"ID":"95abbc15-baab-4701-8410-70a2d6621a5c","Type":"ContainerStarted","Data":"97c7b76893865c85f30f9d9be3d8b7538d4b7ff743d482fcf56ca12552906c5f"}
Sep 05 02:41:14.622592 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.622158    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:14.623865 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.623837    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:41:14.624029 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.623931    2582 generic.go:356] "Generic (PLEG): container finished" podID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerID="b820a8bc85acc4b0d2db38a988295f20aacf5428ca3c04ed8858d3b35de7d6b6" exitCode=2
Sep 05 02:41:14.624029 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.623996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" event={"ID":"a3923ab1-65a6-4eca-80a4-95246afae40e","Type":"ContainerDied","Data":"b820a8bc85acc4b0d2db38a988295f20aacf5428ca3c04ed8858d3b35de7d6b6"}
Sep 05 02:41:14.639798 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:14.639746    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podStartSLOduration=1.639730575 podStartE2EDuration="1.639730575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:14.638687044 +0000 UTC m=+1415.060192384" watchObservedRunningTime="2026-09-05 02:41:14.639730575 +0000 UTC m=+1415.061235914"
Sep 05 02:41:15.628501 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:15.628465    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:41:16.634892 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.634859    2582 generic.go:356] "Generic (PLEG): container finished" podID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerID="0b7eeb587bdd4b413e8b0440d095f6423ef09cdd01a85407ea15a3a4c33cd8b6" exitCode=0
Sep 05 02:41:16.635305 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.634929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" event={"ID":"a3923ab1-65a6-4eca-80a4-95246afae40e","Type":"ContainerDied","Data":"0b7eeb587bdd4b413e8b0440d095f6423ef09cdd01a85407ea15a3a4c33cd8b6"}
Sep 05 02:41:16.721739 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.721716    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:41:16.812469 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.812441    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-rbac-proxy-sar-config\") pod \"a3923ab1-65a6-4eca-80a4-95246afae40e\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") "
Sep 05 02:41:16.812629 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.812520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3923ab1-65a6-4eca-80a4-95246afae40e-proxy-tls\") pod \"a3923ab1-65a6-4eca-80a4-95246afae40e\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") "
Sep 05 02:41:16.812629 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.812543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh9pk\" (UniqueName: \"kubernetes.io/projected/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-api-access-sh9pk\") pod \"a3923ab1-65a6-4eca-80a4-95246afae40e\" (UID: \"a3923ab1-65a6-4eca-80a4-95246afae40e\") "
Sep 05 02:41:16.812873 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.812841    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3923ab1-65a6-4eca-80a4-95246afae40e" (UID: "a3923ab1-65a6-4eca-80a4-95246afae40e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:16.814622 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.814597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3923ab1-65a6-4eca-80a4-95246afae40e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3923ab1-65a6-4eca-80a4-95246afae40e" (UID: "a3923ab1-65a6-4eca-80a4-95246afae40e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:16.814730 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.814682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-api-access-sh9pk" (OuterVolumeSpecName: "kube-api-access-sh9pk") pod "a3923ab1-65a6-4eca-80a4-95246afae40e" (UID: "a3923ab1-65a6-4eca-80a4-95246afae40e"). InnerVolumeSpecName "kube-api-access-sh9pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:16.913805 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.913773    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:41:16.913805 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.913801    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3923ab1-65a6-4eca-80a4-95246afae40e-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:41:16.913805 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:16.913811    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh9pk\" (UniqueName: \"kubernetes.io/projected/a3923ab1-65a6-4eca-80a4-95246afae40e-kube-api-access-sh9pk\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:41:17.641258 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:17.641223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb" event={"ID":"a3923ab1-65a6-4eca-80a4-95246afae40e","Type":"ContainerDied","Data":"b59b9566378830f96bcd2a237a56b4defcce4065c58c8cef31459304b13e8d1b"}
Sep 05 02:41:17.641687 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:17.641277    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"
Sep 05 02:41:17.641687 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:17.641290    2582 scope.go:117] "RemoveContainer" containerID="b820a8bc85acc4b0d2db38a988295f20aacf5428ca3c04ed8858d3b35de7d6b6"
Sep 05 02:41:17.649936 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:17.649915    2582 scope.go:117] "RemoveContainer" containerID="0b7eeb587bdd4b413e8b0440d095f6423ef09cdd01a85407ea15a3a4c33cd8b6"
Sep 05 02:41:17.663778 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:17.663753    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"]
Sep 05 02:41:17.667575 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:17.667553    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9f825-predictor-6cfc7876b6-flxxb"]
Sep 05 02:41:18.109563 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:18.109533    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" path="/var/lib/kubelet/pods/a3923ab1-65a6-4eca-80a4-95246afae40e/volumes"
Sep 05 02:41:20.632879 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:20.632847    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:41:20.633459 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:20.633432    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:41:30.634011 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:30.633955    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:41:33.947561 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:33.947524    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"]
Sep 05 02:41:33.948092 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:33.947857    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container" containerID="cri-o://dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae" gracePeriod=30
Sep 05 02:41:33.948288 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:33.948252    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kube-rbac-proxy" containerID="cri-o://4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0" gracePeriod=30
Sep 05 02:41:34.081340 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081306    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"]
Sep 05 02:41:34.081702 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081688    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container"
Sep 05 02:41:34.081764 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081703    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container"
Sep 05 02:41:34.081764 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081719    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kube-rbac-proxy"
Sep 05 02:41:34.081764 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081725    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kube-rbac-proxy"
Sep 05 02:41:34.081872 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081797    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kube-rbac-proxy"
Sep 05 02:41:34.081872 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.081809    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3923ab1-65a6-4eca-80a4-95246afae40e" containerName="kserve-container"
Sep 05 02:41:34.085173 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.085157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.087923 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.087896    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9755b-kube-rbac-proxy-sar-config\""
Sep 05 02:41:34.088069 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.087896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9755b-predictor-serving-cert\""
Sep 05 02:41:34.093765 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.093743    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"]
Sep 05 02:41:34.164358 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.164324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65rk\" (UniqueName: \"kubernetes.io/projected/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-api-access-r65rk\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.164358 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.164365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.164612 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.164476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.265476 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.265375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65rk\" (UniqueName: \"kubernetes.io/projected/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-api-access-r65rk\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.265476 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.265423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.265725 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.265505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.265725 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:41:34.265681    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9755b-predictor-serving-cert: secret "success-200-isvc-9755b-predictor-serving-cert" not found
Sep 05 02:41:34.265844 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:41:34.265736    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls podName:ef87173c-6ff3-4330-ba4d-05c4c43bd763 nodeName:}" failed. No retries permitted until 2026-09-05 02:41:34.765721449 +0000 UTC m=+1435.187226771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls") pod "success-200-isvc-9755b-predictor-756f474dfd-jjqqk" (UID: "ef87173c-6ff3-4330-ba4d-05c4c43bd763") : secret "success-200-isvc-9755b-predictor-serving-cert" not found
Sep 05 02:41:34.266236 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.266209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.282005 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.281949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65rk\" (UniqueName: \"kubernetes.io/projected/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-api-access-r65rk\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.729398 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.729364    2582 generic.go:356] "Generic (PLEG): container finished" podID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerID="4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0" exitCode=2
Sep 05 02:41:34.729601 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.729431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" event={"ID":"e187ed0e-36e7-49d0-9bd0-141378377e57","Type":"ContainerDied","Data":"4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0"}
Sep 05 02:41:34.770899 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.770860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.773276 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.773250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls\") pod \"success-200-isvc-9755b-predictor-756f474dfd-jjqqk\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") " pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:34.996003 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:34.995879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:35.270692 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.270667    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"]
Sep 05 02:41:35.272604 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:41:35.272574    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef87173c_6ff3_4330_ba4d_05c4c43bd763.slice/crio-118c1136637ebf1f6175770729bb47e7bbbdc70242cb5c562fca36b59a83cd87 WatchSource:0}: Error finding container 118c1136637ebf1f6175770729bb47e7bbbdc70242cb5c562fca36b59a83cd87: Status 404 returned error can't find the container with id 118c1136637ebf1f6175770729bb47e7bbbdc70242cb5c562fca36b59a83cd87
Sep 05 02:41:35.737480 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.737441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" event={"ID":"ef87173c-6ff3-4330-ba4d-05c4c43bd763","Type":"ContainerStarted","Data":"1afef6eef3c00052b432b76ab0685d361ad2377a6932215452921b5c520aad86"}
Sep 05 02:41:35.737713 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.737509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" event={"ID":"ef87173c-6ff3-4330-ba4d-05c4c43bd763","Type":"ContainerStarted","Data":"8259e6a8335341237a6ef05b88b65d62138487062660463d9d591120922da220"}
Sep 05 02:41:35.737713 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.737553    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:35.737713 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.737582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" event={"ID":"ef87173c-6ff3-4330-ba4d-05c4c43bd763","Type":"ContainerStarted","Data":"118c1136637ebf1f6175770729bb47e7bbbdc70242cb5c562fca36b59a83cd87"}
Sep 05 02:41:35.737713 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.737612    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:35.739139 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.739109    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:41:35.759847 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:35.759787    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podStartSLOduration=1.759772108 podStartE2EDuration="1.759772108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:35.757745711 +0000 UTC m=+1436.179251077" watchObservedRunningTime="2026-09-05 02:41:35.759772108 +0000 UTC m=+1436.181277447"
Sep 05 02:41:36.748697 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:36.748646    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:41:37.199053 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.199029    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:41:37.293296 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.293267    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-rbac-proxy-sar-config\") pod \"e187ed0e-36e7-49d0-9bd0-141378377e57\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") "
Sep 05 02:41:37.293485 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.293336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbtmk\" (UniqueName: \"kubernetes.io/projected/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-api-access-pbtmk\") pod \"e187ed0e-36e7-49d0-9bd0-141378377e57\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") "
Sep 05 02:41:37.293485 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.293440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls\") pod \"e187ed0e-36e7-49d0-9bd0-141378377e57\" (UID: \"e187ed0e-36e7-49d0-9bd0-141378377e57\") "
Sep 05 02:41:37.293705 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.293675    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e187ed0e-36e7-49d0-9bd0-141378377e57" (UID: "e187ed0e-36e7-49d0-9bd0-141378377e57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:37.293805 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.293790    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:41:37.295276 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.295254    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e187ed0e-36e7-49d0-9bd0-141378377e57" (UID: "e187ed0e-36e7-49d0-9bd0-141378377e57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:37.295393 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.295373    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-api-access-pbtmk" (OuterVolumeSpecName: "kube-api-access-pbtmk") pod "e187ed0e-36e7-49d0-9bd0-141378377e57" (UID: "e187ed0e-36e7-49d0-9bd0-141378377e57"). InnerVolumeSpecName "kube-api-access-pbtmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:37.394490 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.394450    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtmk\" (UniqueName: \"kubernetes.io/projected/e187ed0e-36e7-49d0-9bd0-141378377e57-kube-api-access-pbtmk\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:41:37.394490 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.394481    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e187ed0e-36e7-49d0-9bd0-141378377e57-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:41:37.751048 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.751018    2582 generic.go:356] "Generic (PLEG): container finished" podID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerID="dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae" exitCode=0
Sep 05 02:41:37.751514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.751085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" event={"ID":"e187ed0e-36e7-49d0-9bd0-141378377e57","Type":"ContainerDied","Data":"dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae"}
Sep 05 02:41:37.751514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.751091    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"
Sep 05 02:41:37.751514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.751193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm" event={"ID":"e187ed0e-36e7-49d0-9bd0-141378377e57","Type":"ContainerDied","Data":"fb3092fb29395cbb29fdbeb4146b731544acf6fe35bdd4012042c636747ebc7d"}
Sep 05 02:41:37.751514 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.751234    2582 scope.go:117] "RemoveContainer" containerID="4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0"
Sep 05 02:41:37.760530 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.760449    2582 scope.go:117] "RemoveContainer" containerID="dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae"
Sep 05 02:41:37.768057 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.768040    2582 scope.go:117] "RemoveContainer" containerID="4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0"
Sep 05 02:41:37.768323 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:41:37.768304    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0\": container with ID starting with 4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0 not found: ID does not exist" containerID="4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0"
Sep 05 02:41:37.768380 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.768331    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0"} err="failed to get container status \"4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0\": rpc error: code = NotFound desc = could not find container \"4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0\": container with ID starting with 4cc91704b994516b28d2e70ffb5ca13c38027c6bf087644962327069397903b0 not found: ID does not exist"
Sep 05 02:41:37.768380 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.768347    2582 scope.go:117] "RemoveContainer" containerID="dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae"
Sep 05 02:41:37.768577 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:41:37.768559    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae\": container with ID starting with dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae not found: ID does not exist" containerID="dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae"
Sep 05 02:41:37.768637 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.768587    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae"} err="failed to get container status \"dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae\": rpc error: code = NotFound desc = could not find container \"dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae\": container with ID starting with dff60d54814a8b4f07bcef47cecc558f4c420d35c7e8d0f92af4cac6c33357ae not found: ID does not exist"
Sep 05 02:41:37.787620 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.787417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"]
Sep 05 02:41:37.789302 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:37.789281    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12d88-predictor-7b774fdf4d-mp9lm"]
Sep 05 02:41:38.109837 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:38.109804    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" path="/var/lib/kubelet/pods/e187ed0e-36e7-49d0-9bd0-141378377e57/volumes"
Sep 05 02:41:40.633759 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:40.633724    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:41:41.749264 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:41.749234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:41:41.749852 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:41.749814    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:41:50.634191 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:50.634150    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:41:51.750271 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:41:51.750233    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:42:00.635092 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:00.635057    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:42:01.749881 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:01.749837    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:42:11.749764 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:11.749709    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:42:21.750324 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:21.750286    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:42:23.875452 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:23.875418    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"]
Sep 05 02:42:23.875854 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:23.875676    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container" containerID="cri-o://584e47f6ae9254bd32ab03e30f386928f9d710775d122a0709060ba044c538fa" gracePeriod=30
Sep 05 02:42:23.875854 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:23.875715    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kube-rbac-proxy" containerID="cri-o://9d5ba700ca9581b0b4962fe02a4d7af7f3f48aedc72cc29f19accb91997acaca" gracePeriod=30
Sep 05 02:42:24.011928 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.011883    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"]
Sep 05 02:42:24.012753 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.012726    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kube-rbac-proxy"
Sep 05 02:42:24.012753 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.012752    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kube-rbac-proxy"
Sep 05 02:42:24.012934 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.012771    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container"
Sep 05 02:42:24.012934 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.012779    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container"
Sep 05 02:42:24.012934 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.012866    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kserve-container"
Sep 05 02:42:24.012934 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.012877    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e187ed0e-36e7-49d0-9bd0-141378377e57" containerName="kube-rbac-proxy"
Sep 05 02:42:24.016524 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.016504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.019696 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.019671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-43d66-predictor-serving-cert\""
Sep 05 02:42:24.019817 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.019709    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-43d66-kube-rbac-proxy-sar-config\""
Sep 05 02:42:24.026414 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.026385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"]
Sep 05 02:42:24.111043 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.111014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjll\" (UniqueName: \"kubernetes.io/projected/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-api-access-lfjll\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.111206 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.111061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.111206 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.111087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.212570 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.212485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.212570 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.212524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.212807 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.212597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjll\" (UniqueName: \"kubernetes.io/projected/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-api-access-lfjll\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.212807 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:42:24.212657    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-43d66-predictor-serving-cert: secret "success-200-isvc-43d66-predictor-serving-cert" not found
Sep 05 02:42:24.212807 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:42:24.212744    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls podName:db614bb6-145c-4949-a2f6-4953b589f6b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:42:24.712723476 +0000 UTC m=+1485.134228810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls") pod "success-200-isvc-43d66-predictor-7ff977846d-4qxhj" (UID: "db614bb6-145c-4949-a2f6-4953b589f6b5") : secret "success-200-isvc-43d66-predictor-serving-cert" not found
Sep 05 02:42:24.213286 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.213267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.223782 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.223756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjll\" (UniqueName: \"kubernetes.io/projected/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-api-access-lfjll\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.717758 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.717719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:24.717940 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:42:24.717866    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-43d66-predictor-serving-cert: secret "success-200-isvc-43d66-predictor-serving-cert" not found
Sep 05 02:42:24.717940 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:42:24.717929    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls podName:db614bb6-145c-4949-a2f6-4953b589f6b5 nodeName:}" failed. No retries permitted until 2026-09-05 02:42:25.717914686 +0000 UTC m=+1486.139420002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls") pod "success-200-isvc-43d66-predictor-7ff977846d-4qxhj" (UID: "db614bb6-145c-4949-a2f6-4953b589f6b5") : secret "success-200-isvc-43d66-predictor-serving-cert" not found
Sep 05 02:42:24.984141 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.984060    2582 generic.go:356] "Generic (PLEG): container finished" podID="95abbc15-baab-4701-8410-70a2d6621a5c" containerID="9d5ba700ca9581b0b4962fe02a4d7af7f3f48aedc72cc29f19accb91997acaca" exitCode=2
Sep 05 02:42:24.984141 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:24.984129    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" event={"ID":"95abbc15-baab-4701-8410-70a2d6621a5c","Type":"ContainerDied","Data":"9d5ba700ca9581b0b4962fe02a4d7af7f3f48aedc72cc29f19accb91997acaca"}
Sep 05 02:42:25.629376 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:25.629338    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 05 02:42:25.728155 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:25.728117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:25.730659 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:25.730628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") pod \"success-200-isvc-43d66-predictor-7ff977846d-4qxhj\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") " pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:25.828467 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:25.828431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:26.110426 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:26.110398    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"]
Sep 05 02:42:26.113763 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:42:26.113734    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb614bb6_145c_4949_a2f6_4953b589f6b5.slice/crio-ecf55bbe99fce4207378706c8830b0ff7f64d6765bfb4e8e3d6b1cbe637cc3a9 WatchSource:0}: Error finding container ecf55bbe99fce4207378706c8830b0ff7f64d6765bfb4e8e3d6b1cbe637cc3a9: Status 404 returned error can't find the container with id ecf55bbe99fce4207378706c8830b0ff7f64d6765bfb4e8e3d6b1cbe637cc3a9
Sep 05 02:42:26.998239 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:26.998202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" event={"ID":"db614bb6-145c-4949-a2f6-4953b589f6b5","Type":"ContainerStarted","Data":"655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd"}
Sep 05 02:42:26.998426 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:26.998258    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" event={"ID":"db614bb6-145c-4949-a2f6-4953b589f6b5","Type":"ContainerStarted","Data":"11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f"}
Sep 05 02:42:26.998426 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:26.998286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" event={"ID":"db614bb6-145c-4949-a2f6-4953b589f6b5","Type":"ContainerStarted","Data":"ecf55bbe99fce4207378706c8830b0ff7f64d6765bfb4e8e3d6b1cbe637cc3a9"}
Sep 05 02:42:26.998426 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:26.998318    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:27.000257 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.000229    2582 generic.go:356] "Generic (PLEG): container finished" podID="95abbc15-baab-4701-8410-70a2d6621a5c" containerID="584e47f6ae9254bd32ab03e30f386928f9d710775d122a0709060ba044c538fa" exitCode=0
Sep 05 02:42:27.000371 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.000278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" event={"ID":"95abbc15-baab-4701-8410-70a2d6621a5c","Type":"ContainerDied","Data":"584e47f6ae9254bd32ab03e30f386928f9d710775d122a0709060ba044c538fa"}
Sep 05 02:42:27.017070 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.017023    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podStartSLOduration=4.017005791 podStartE2EDuration="4.017005791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:42:27.015739754 +0000 UTC m=+1487.437245094" watchObservedRunningTime="2026-09-05 02:42:27.017005791 +0000 UTC m=+1487.438511129"
Sep 05 02:42:27.030812 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.030784    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:42:27.142089 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.141987    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95abbc15-baab-4701-8410-70a2d6621a5c-kube-rbac-proxy-sar-config\") pod \"95abbc15-baab-4701-8410-70a2d6621a5c\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") "
Sep 05 02:42:27.142539 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.142134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95abbc15-baab-4701-8410-70a2d6621a5c-proxy-tls\") pod \"95abbc15-baab-4701-8410-70a2d6621a5c\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") "
Sep 05 02:42:27.142539 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.142223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzt6w\" (UniqueName: \"kubernetes.io/projected/95abbc15-baab-4701-8410-70a2d6621a5c-kube-api-access-zzt6w\") pod \"95abbc15-baab-4701-8410-70a2d6621a5c\" (UID: \"95abbc15-baab-4701-8410-70a2d6621a5c\") "
Sep 05 02:42:27.142539 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.142377    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95abbc15-baab-4701-8410-70a2d6621a5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95abbc15-baab-4701-8410-70a2d6621a5c" (UID: "95abbc15-baab-4701-8410-70a2d6621a5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:27.142667 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.142550    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95abbc15-baab-4701-8410-70a2d6621a5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:42:27.144370 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.144346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95abbc15-baab-4701-8410-70a2d6621a5c-kube-api-access-zzt6w" (OuterVolumeSpecName: "kube-api-access-zzt6w") pod "95abbc15-baab-4701-8410-70a2d6621a5c" (UID: "95abbc15-baab-4701-8410-70a2d6621a5c"). InnerVolumeSpecName "kube-api-access-zzt6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:27.144455 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.144382    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95abbc15-baab-4701-8410-70a2d6621a5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95abbc15-baab-4701-8410-70a2d6621a5c" (UID: "95abbc15-baab-4701-8410-70a2d6621a5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:27.243724 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.243683    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95abbc15-baab-4701-8410-70a2d6621a5c-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:42:27.243724 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:27.243715    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzt6w\" (UniqueName: \"kubernetes.io/projected/95abbc15-baab-4701-8410-70a2d6621a5c-kube-api-access-zzt6w\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:42:28.008764 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.008736    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"
Sep 05 02:42:28.008951 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.008763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5" event={"ID":"95abbc15-baab-4701-8410-70a2d6621a5c","Type":"ContainerDied","Data":"97c7b76893865c85f30f9d9be3d8b7538d4b7ff743d482fcf56ca12552906c5f"}
Sep 05 02:42:28.008951 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.008825    2582 scope.go:117] "RemoveContainer" containerID="9d5ba700ca9581b0b4962fe02a4d7af7f3f48aedc72cc29f19accb91997acaca"
Sep 05 02:42:28.009287 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.009266    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:28.011047 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.010935    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:28.017848 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.017833    2582 scope.go:117] "RemoveContainer" containerID="584e47f6ae9254bd32ab03e30f386928f9d710775d122a0709060ba044c538fa"
Sep 05 02:42:28.031525 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.031500    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"]
Sep 05 02:42:28.037554 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.037531    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80c39-predictor-59cf7d7b96-xp9n5"]
Sep 05 02:42:28.110060 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:28.110028    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" path="/var/lib/kubelet/pods/95abbc15-baab-4701-8410-70a2d6621a5c/volumes"
Sep 05 02:42:29.015157 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:29.015125    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:34.020132 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:34.020100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:42:34.020513 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:34.020482    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:44.020570 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:44.020532    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:42:54.021448 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:42:54.021407    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:43:04.021369 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:43:04.021284    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 05 02:43:14.021142 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:43:14.021109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:50:49.053887 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.053800    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"]
Sep 05 02:50:49.054538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.054319    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container"
Sep 05 02:50:49.054538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.054335    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container"
Sep 05 02:50:49.054538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.054352    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kube-rbac-proxy"
Sep 05 02:50:49.054538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.054358    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kube-rbac-proxy"
Sep 05 02:50:49.054538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.054434    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kserve-container"
Sep 05 02:50:49.054538 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.054449    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95abbc15-baab-4701-8410-70a2d6621a5c" containerName="kube-rbac-proxy"
Sep 05 02:50:49.057817 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.057795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.060658 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.060630    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ca9bf-predictor-serving-cert\""
Sep 05 02:50:49.060852 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.060835    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ca9bf-kube-rbac-proxy-sar-config\""
Sep 05 02:50:49.068537 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.068503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"]
Sep 05 02:50:49.072551 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.072525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"]
Sep 05 02:50:49.072804 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.072772    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" containerID="cri-o://8259e6a8335341237a6ef05b88b65d62138487062660463d9d591120922da220" gracePeriod=30
Sep 05 02:50:49.073040 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.072994    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kube-rbac-proxy" containerID="cri-o://1afef6eef3c00052b432b76ab0685d361ad2377a6932215452921b5c520aad86" gracePeriod=30
Sep 05 02:50:49.174265 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.174227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7gj\" (UniqueName: \"kubernetes.io/projected/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-api-access-dl7gj\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.174449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.174287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.174449 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.174316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.275308 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.275263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7gj\" (UniqueName: \"kubernetes.io/projected/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-api-access-dl7gj\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.275459 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.275348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.275459 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.275380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.275573 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:50:49.275457    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-serving-cert: secret "success-200-isvc-ca9bf-predictor-serving-cert" not found
Sep 05 02:50:49.275573 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:50:49.275537    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls podName:6a5b59a7-f1d9-4886-95ff-48f213b8ea08 nodeName:}" failed. No retries permitted until 2026-09-05 02:50:49.775517139 +0000 UTC m=+1990.197022456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls") pod "success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" (UID: "6a5b59a7-f1d9-4886-95ff-48f213b8ea08") : secret "success-200-isvc-ca9bf-predictor-serving-cert" not found
Sep 05 02:50:49.276213 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.276191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.284620 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.284588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7gj\" (UniqueName: \"kubernetes.io/projected/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-api-access-dl7gj\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.368086 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.368053    2582 generic.go:356] "Generic (PLEG): container finished" podID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerID="1afef6eef3c00052b432b76ab0685d361ad2377a6932215452921b5c520aad86" exitCode=2
Sep 05 02:50:49.368278 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.368131    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" event={"ID":"ef87173c-6ff3-4330-ba4d-05c4c43bd763","Type":"ContainerDied","Data":"1afef6eef3c00052b432b76ab0685d361ad2377a6932215452921b5c520aad86"}
Sep 05 02:50:49.779960 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.779866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.782278 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.782251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls\") pod \"success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") " pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:49.970109 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:49.970064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:50.251466 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:50.251434    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"]
Sep 05 02:50:50.253232 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:50:50.253206    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5b59a7_f1d9_4886_95ff_48f213b8ea08.slice/crio-8a026156a0104b3e8c889fd49ee106710ae5c9cc792c02a98aff01e9092cb155 WatchSource:0}: Error finding container 8a026156a0104b3e8c889fd49ee106710ae5c9cc792c02a98aff01e9092cb155: Status 404 returned error can't find the container with id 8a026156a0104b3e8c889fd49ee106710ae5c9cc792c02a98aff01e9092cb155
Sep 05 02:50:50.375196 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:50.375168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" event={"ID":"6a5b59a7-f1d9-4886-95ff-48f213b8ea08","Type":"ContainerStarted","Data":"dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8"}
Sep 05 02:50:50.375336 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:50.375225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" event={"ID":"6a5b59a7-f1d9-4886-95ff-48f213b8ea08","Type":"ContainerStarted","Data":"8a026156a0104b3e8c889fd49ee106710ae5c9cc792c02a98aff01e9092cb155"}
Sep 05 02:50:51.382394 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.382354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" event={"ID":"6a5b59a7-f1d9-4886-95ff-48f213b8ea08","Type":"ContainerStarted","Data":"5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6"}
Sep 05 02:50:51.382822 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.382602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:51.382822 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.382745    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:51.384110 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.384082    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:50:51.400797 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.400750    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podStartSLOduration=2.400731979 podStartE2EDuration="2.400731979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:50:51.400082402 +0000 UTC m=+1991.821587752" watchObservedRunningTime="2026-09-05 02:50:51.400731979 +0000 UTC m=+1991.822237318"
Sep 05 02:50:51.746456 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.746342    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 05 02:50:51.749844 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:51.749805    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 05 02:50:52.391052 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.391014    2582 generic.go:356] "Generic (PLEG): container finished" podID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerID="8259e6a8335341237a6ef05b88b65d62138487062660463d9d591120922da220" exitCode=0
Sep 05 02:50:52.391424 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.391083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" event={"ID":"ef87173c-6ff3-4330-ba4d-05c4c43bd763","Type":"ContainerDied","Data":"8259e6a8335341237a6ef05b88b65d62138487062660463d9d591120922da220"}
Sep 05 02:50:52.391482 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.391466    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:50:52.431073 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.431046    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:50:52.607231 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.607198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r65rk\" (UniqueName: \"kubernetes.io/projected/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-api-access-r65rk\") pod \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") "
Sep 05 02:50:52.607395 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.607245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-rbac-proxy-sar-config\") pod \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") "
Sep 05 02:50:52.607395 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.607299    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls\") pod \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\" (UID: \"ef87173c-6ff3-4330-ba4d-05c4c43bd763\") "
Sep 05 02:50:52.607636 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.607600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef87173c-6ff3-4330-ba4d-05c4c43bd763" (UID: "ef87173c-6ff3-4330-ba4d-05c4c43bd763"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:50:52.609360 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.609334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef87173c-6ff3-4330-ba4d-05c4c43bd763" (UID: "ef87173c-6ff3-4330-ba4d-05c4c43bd763"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:52.609436 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.609344    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-api-access-r65rk" (OuterVolumeSpecName: "kube-api-access-r65rk") pod "ef87173c-6ff3-4330-ba4d-05c4c43bd763" (UID: "ef87173c-6ff3-4330-ba4d-05c4c43bd763"). InnerVolumeSpecName "kube-api-access-r65rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:52.708366 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.708326    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87173c-6ff3-4330-ba4d-05c4c43bd763-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:50:52.708366 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.708357    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65rk\" (UniqueName: \"kubernetes.io/projected/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-api-access-r65rk\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:50:52.708366 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:52.708369    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87173c-6ff3-4330-ba4d-05c4c43bd763-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:50:53.398150 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:53.398124    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"
Sep 05 02:50:53.398558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:53.398127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk" event={"ID":"ef87173c-6ff3-4330-ba4d-05c4c43bd763","Type":"ContainerDied","Data":"118c1136637ebf1f6175770729bb47e7bbbdc70242cb5c562fca36b59a83cd87"}
Sep 05 02:50:53.398558 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:53.398267    2582 scope.go:117] "RemoveContainer" containerID="1afef6eef3c00052b432b76ab0685d361ad2377a6932215452921b5c520aad86"
Sep 05 02:50:53.407158 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:53.407137    2582 scope.go:117] "RemoveContainer" containerID="8259e6a8335341237a6ef05b88b65d62138487062660463d9d591120922da220"
Sep 05 02:50:53.420000 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:53.419958    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"]
Sep 05 02:50:53.421598 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:53.421579    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9755b-predictor-756f474dfd-jjqqk"]
Sep 05 02:50:54.109470 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:54.109437    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" path="/var/lib/kubelet/pods/ef87173c-6ff3-4330-ba4d-05c4c43bd763/volumes"
Sep 05 02:50:57.396011 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:57.395958    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:50:57.396521 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:50:57.396494    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:51:07.396943 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:07.396902    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:51:17.396536 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:17.396487    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:51:27.396937 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:27.396887    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:51:37.398166 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:37.398133    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:51:48.806672 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:48.806636    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"]
Sep 05 02:51:48.807102 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:48.806879    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container" containerID="cri-o://11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f" gracePeriod=30
Sep 05 02:51:48.807102 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:48.806956    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kube-rbac-proxy" containerID="cri-o://655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd" gracePeriod=30
Sep 05 02:51:49.008207 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008174    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"]
Sep 05 02:51:49.008560 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008547    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container"
Sep 05 02:51:49.008603 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008561    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container"
Sep 05 02:51:49.008603 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008577    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kube-rbac-proxy"
Sep 05 02:51:49.008603 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008583    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kube-rbac-proxy"
Sep 05 02:51:49.008696 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008644    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kube-rbac-proxy"
Sep 05 02:51:49.008696 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.008652    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef87173c-6ff3-4330-ba4d-05c4c43bd763" containerName="kserve-container"
Sep 05 02:51:49.011906 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.011886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.014547 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.014522    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3af6d-predictor-serving-cert\""
Sep 05 02:51:49.014664 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.014567    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3af6d-kube-rbac-proxy-sar-config\""
Sep 05 02:51:49.016175 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.016148    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 05 02:51:49.025130 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.025101    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"]
Sep 05 02:51:49.101194 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.101160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96js\" (UniqueName: \"kubernetes.io/projected/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-api-access-d96js\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.101377 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.101262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.101377 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.101288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.202142 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.202106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96js\" (UniqueName: \"kubernetes.io/projected/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-api-access-d96js\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.202314 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.202267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.202314 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.202308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.202472 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:51:49.202458    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-serving-cert: secret "success-200-isvc-3af6d-predictor-serving-cert" not found
Sep 05 02:51:49.202527 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:51:49.202511    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls podName:a7271d07-a22c-42cb-9e19-6587036c8eff nodeName:}" failed. No retries permitted until 2026-09-05 02:51:49.702497328 +0000 UTC m=+2050.124002644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls") pod "success-200-isvc-3af6d-predictor-67876986cd-bjgx2" (UID: "a7271d07-a22c-42cb-9e19-6587036c8eff") : secret "success-200-isvc-3af6d-predictor-serving-cert" not found
Sep 05 02:51:49.202897 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.202878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.212574 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.212552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96js\" (UniqueName: \"kubernetes.io/projected/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-api-access-d96js\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.663414 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.663379    2582 generic.go:356] "Generic (PLEG): container finished" podID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerID="655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd" exitCode=2
Sep 05 02:51:49.663593 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.663452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" event={"ID":"db614bb6-145c-4949-a2f6-4953b589f6b5","Type":"ContainerDied","Data":"655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd"}
Sep 05 02:51:49.705936 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.705896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.708491 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.708467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls\") pod \"success-200-isvc-3af6d-predictor-67876986cd-bjgx2\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") " pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:49.923457 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:49.923357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:50.196799 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:50.196757    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"]
Sep 05 02:51:50.199103 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:51:50.199073    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7271d07_a22c_42cb_9e19_6587036c8eff.slice/crio-940c9ac085bf535690dacdf864cae8d1534f86ca674328f1dcbc44a9dbfe4b7d WatchSource:0}: Error finding container 940c9ac085bf535690dacdf864cae8d1534f86ca674328f1dcbc44a9dbfe4b7d: Status 404 returned error can't find the container with id 940c9ac085bf535690dacdf864cae8d1534f86ca674328f1dcbc44a9dbfe4b7d
Sep 05 02:51:50.670319 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:50.670280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" event={"ID":"a7271d07-a22c-42cb-9e19-6587036c8eff","Type":"ContainerStarted","Data":"b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8"}
Sep 05 02:51:50.670520 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:50.670347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" event={"ID":"a7271d07-a22c-42cb-9e19-6587036c8eff","Type":"ContainerStarted","Data":"b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33"}
Sep 05 02:51:50.670520 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:50.670369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" event={"ID":"a7271d07-a22c-42cb-9e19-6587036c8eff","Type":"ContainerStarted","Data":"940c9ac085bf535690dacdf864cae8d1534f86ca674328f1dcbc44a9dbfe4b7d"}
Sep 05 02:51:50.670520 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:50.670408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:50.689018 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:50.688944    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podStartSLOduration=2.688930057 podStartE2EDuration="2.688930057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:50.688362413 +0000 UTC m=+2051.109867753" watchObservedRunningTime="2026-09-05 02:51:50.688930057 +0000 UTC m=+2051.110435395"
Sep 05 02:51:51.675427 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:51.675396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:51.676853 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:51.676821    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:51:52.167363 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.167331    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:51:52.227252 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.227168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") pod \"db614bb6-145c-4949-a2f6-4953b589f6b5\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") "
Sep 05 02:51:52.227252 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.227213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfjll\" (UniqueName: \"kubernetes.io/projected/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-api-access-lfjll\") pod \"db614bb6-145c-4949-a2f6-4953b589f6b5\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") "
Sep 05 02:51:52.227252 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.227243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-rbac-proxy-sar-config\") pod \"db614bb6-145c-4949-a2f6-4953b589f6b5\" (UID: \"db614bb6-145c-4949-a2f6-4953b589f6b5\") "
Sep 05 02:51:52.227634 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.227612    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db614bb6-145c-4949-a2f6-4953b589f6b5" (UID: "db614bb6-145c-4949-a2f6-4953b589f6b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:51:52.229350 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.229325    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-api-access-lfjll" (OuterVolumeSpecName: "kube-api-access-lfjll") pod "db614bb6-145c-4949-a2f6-4953b589f6b5" (UID: "db614bb6-145c-4949-a2f6-4953b589f6b5"). InnerVolumeSpecName "kube-api-access-lfjll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:51:52.229439 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.229352    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db614bb6-145c-4949-a2f6-4953b589f6b5" (UID: "db614bb6-145c-4949-a2f6-4953b589f6b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:51:52.328202 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.328157    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db614bb6-145c-4949-a2f6-4953b589f6b5-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:51:52.328202 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.328200    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjll\" (UniqueName: \"kubernetes.io/projected/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-api-access-lfjll\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:51:52.328202 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.328210    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db614bb6-145c-4949-a2f6-4953b589f6b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:51:52.681646 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.681609    2582 generic.go:356] "Generic (PLEG): container finished" podID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerID="11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f" exitCode=0
Sep 05 02:51:52.682087 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.681662    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"
Sep 05 02:51:52.682087 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.681669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" event={"ID":"db614bb6-145c-4949-a2f6-4953b589f6b5","Type":"ContainerDied","Data":"11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f"}
Sep 05 02:51:52.682087 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.681736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj" event={"ID":"db614bb6-145c-4949-a2f6-4953b589f6b5","Type":"ContainerDied","Data":"ecf55bbe99fce4207378706c8830b0ff7f64d6765bfb4e8e3d6b1cbe637cc3a9"}
Sep 05 02:51:52.682087 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.681772    2582 scope.go:117] "RemoveContainer" containerID="655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd"
Sep 05 02:51:52.682325 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.682294    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:51:52.691364 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.691343    2582 scope.go:117] "RemoveContainer" containerID="11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f"
Sep 05 02:51:52.699417 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.699399    2582 scope.go:117] "RemoveContainer" containerID="655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd"
Sep 05 02:51:52.699675 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:51:52.699655    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd\": container with ID starting with 655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd not found: ID does not exist" containerID="655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd"
Sep 05 02:51:52.699728 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.699685    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd"} err="failed to get container status \"655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd\": rpc error: code = NotFound desc = could not find container \"655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd\": container with ID starting with 655e64dfc33200d89f07c857b85e4b2b42acf23f35562bd89d13a531917c3cdd not found: ID does not exist"
Sep 05 02:51:52.699728 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.699711    2582 scope.go:117] "RemoveContainer" containerID="11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f"
Sep 05 02:51:52.699960 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:51:52.699942    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f\": container with ID starting with 11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f not found: ID does not exist" containerID="11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f"
Sep 05 02:51:52.700028 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.699986    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f"} err="failed to get container status \"11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f\": rpc error: code = NotFound desc = could not find container \"11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f\": container with ID starting with 11f8185b5fa19c596d4d39d26f8ddc3a76a0dea5a39ba49d4eddb79e3d477f8f not found: ID does not exist"
Sep 05 02:51:52.704353 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.704324    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"]
Sep 05 02:51:52.706161 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:52.706141    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43d66-predictor-7ff977846d-4qxhj"]
Sep 05 02:51:54.109865 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:54.109831    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" path="/var/lib/kubelet/pods/db614bb6-145c-4949-a2f6-4953b589f6b5/volumes"
Sep 05 02:51:57.687248 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:57.687215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:51:57.687731 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:57.687697    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:51:59.244257 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.244181    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"]
Sep 05 02:51:59.244663 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.244403    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container" containerID="cri-o://dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8" gracePeriod=30
Sep 05 02:51:59.244663 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.244460    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kube-rbac-proxy" containerID="cri-o://5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6" gracePeriod=30
Sep 05 02:51:59.379170 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379133    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"]
Sep 05 02:51:59.379519 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379507    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container"
Sep 05 02:51:59.379575 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379521    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container"
Sep 05 02:51:59.379575 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379537    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kube-rbac-proxy"
Sep 05 02:51:59.379575 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379543    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kube-rbac-proxy"
Sep 05 02:51:59.379673 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379591    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kube-rbac-proxy"
Sep 05 02:51:59.379673 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.379602    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db614bb6-145c-4949-a2f6-4953b589f6b5" containerName="kserve-container"
Sep 05 02:51:59.382816 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.382793    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.385162 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.385138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a4a3e-predictor-serving-cert\""
Sep 05 02:51:59.385300 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.385283    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a4a3e-kube-rbac-proxy-sar-config\""
Sep 05 02:51:59.396482 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.396456    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"]
Sep 05 02:51:59.488279 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.488244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-proxy-tls\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.488443 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.488288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.488443 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.488358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjdr\" (UniqueName: \"kubernetes.io/projected/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-api-access-xnjdr\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.589516 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.589477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjdr\" (UniqueName: \"kubernetes.io/projected/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-api-access-xnjdr\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.589708 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.589554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-proxy-tls\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.589708 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.589585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.590357 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.590333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.592069 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.592049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-proxy-tls\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.597619 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.597595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjdr\" (UniqueName: \"kubernetes.io/projected/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-api-access-xnjdr\") pod \"success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") " pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.693363 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.693321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:51:59.723007 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.722949    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerID="5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6" exitCode=2
Sep 05 02:51:59.723007 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.722998    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" event={"ID":"6a5b59a7-f1d9-4886-95ff-48f213b8ea08","Type":"ContainerDied","Data":"5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6"}
Sep 05 02:51:59.967664 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:51:59.967638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"]
Sep 05 02:51:59.969338 ip-10-0-138-105 kubenswrapper[2582]: W0905 02:51:59.969307    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb1ea22_a856_4fd3_bfaa_22335d14af5d.slice/crio-1345ee8aa024a84b139bd56b0c94bc12b636ed77df5fffb2f5d9df8cca886f28 WatchSource:0}: Error finding container 1345ee8aa024a84b139bd56b0c94bc12b636ed77df5fffb2f5d9df8cca886f28: Status 404 returned error can't find the container with id 1345ee8aa024a84b139bd56b0c94bc12b636ed77df5fffb2f5d9df8cca886f28
Sep 05 02:52:00.729713 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.729673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" event={"ID":"2eb1ea22-a856-4fd3-bfaa-22335d14af5d","Type":"ContainerStarted","Data":"7b507c213df2021e03d9033e49ba9be4be735a4b3a3e7e4eb2e3c1cd40512d9a"}
Sep 05 02:52:00.730153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.729744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" event={"ID":"2eb1ea22-a856-4fd3-bfaa-22335d14af5d","Type":"ContainerStarted","Data":"58ab0a7e91a86f1246aff6d509a3bad7db487842abe9f1ed13c900173d9445be"}
Sep 05 02:52:00.730153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.729864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:52:00.730153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.729903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" event={"ID":"2eb1ea22-a856-4fd3-bfaa-22335d14af5d","Type":"ContainerStarted","Data":"1345ee8aa024a84b139bd56b0c94bc12b636ed77df5fffb2f5d9df8cca886f28"}
Sep 05 02:52:00.730153 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.729927    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:52:00.731518 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.731489    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:52:00.747930 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:00.747886    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podStartSLOduration=1.747868183 podStartE2EDuration="1.747868183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:52:00.747127018 +0000 UTC m=+2061.168632382" watchObservedRunningTime="2026-09-05 02:52:00.747868183 +0000 UTC m=+2061.169373585"
Sep 05 02:52:01.739632 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:01.739584    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:52:02.392062 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.392022    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 05 02:52:02.596441 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.596415    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:52:02.616482 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.616390    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl7gj\" (UniqueName: \"kubernetes.io/projected/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-api-access-dl7gj\") pod \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") "
Sep 05 02:52:02.616482 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.616459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls\") pod \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") "
Sep 05 02:52:02.616687 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.616576    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-rbac-proxy-sar-config\") pod \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\" (UID: \"6a5b59a7-f1d9-4886-95ff-48f213b8ea08\") "
Sep 05 02:52:02.617102 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.617073    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a5b59a7-f1d9-4886-95ff-48f213b8ea08" (UID: "6a5b59a7-f1d9-4886-95ff-48f213b8ea08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:52:02.618703 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.618656    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a5b59a7-f1d9-4886-95ff-48f213b8ea08" (UID: "6a5b59a7-f1d9-4886-95ff-48f213b8ea08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:52:02.618839 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.618820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-api-access-dl7gj" (OuterVolumeSpecName: "kube-api-access-dl7gj") pod "6a5b59a7-f1d9-4886-95ff-48f213b8ea08" (UID: "6a5b59a7-f1d9-4886-95ff-48f213b8ea08"). InnerVolumeSpecName "kube-api-access-dl7gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:52:02.717845 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.717810    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:52:02.717845 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.717839    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7gj\" (UniqueName: \"kubernetes.io/projected/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-kube-api-access-dl7gj\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:52:02.717845 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.717851    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5b59a7-f1d9-4886-95ff-48f213b8ea08-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 02:52:02.741463 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.741428    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerID="dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8" exitCode=0
Sep 05 02:52:02.741875 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.741478    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"
Sep 05 02:52:02.741875 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.741492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" event={"ID":"6a5b59a7-f1d9-4886-95ff-48f213b8ea08","Type":"ContainerDied","Data":"dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8"}
Sep 05 02:52:02.741875 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.741551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc" event={"ID":"6a5b59a7-f1d9-4886-95ff-48f213b8ea08","Type":"ContainerDied","Data":"8a026156a0104b3e8c889fd49ee106710ae5c9cc792c02a98aff01e9092cb155"}
Sep 05 02:52:02.741875 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.741577    2582 scope.go:117] "RemoveContainer" containerID="5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6"
Sep 05 02:52:02.751989 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.751942    2582 scope.go:117] "RemoveContainer" containerID="dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8"
Sep 05 02:52:02.759812 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.759793    2582 scope.go:117] "RemoveContainer" containerID="5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6"
Sep 05 02:52:02.760113 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:52:02.760095    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6\": container with ID starting with 5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6 not found: ID does not exist" containerID="5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6"
Sep 05 02:52:02.760169 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.760123    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6"} err="failed to get container status \"5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6\": rpc error: code = NotFound desc = could not find container \"5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6\": container with ID starting with 5f85014979533ae6d474b33f507be6ce100afca19e4fd6506e02ec8d87acb3a6 not found: ID does not exist"
Sep 05 02:52:02.760169 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.760141    2582 scope.go:117] "RemoveContainer" containerID="dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8"
Sep 05 02:52:02.760419 ip-10-0-138-105 kubenswrapper[2582]: E0905 02:52:02.760401    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8\": container with ID starting with dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8 not found: ID does not exist" containerID="dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8"
Sep 05 02:52:02.760468 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.760423    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8"} err="failed to get container status \"dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8\": rpc error: code = NotFound desc = could not find container \"dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8\": container with ID starting with dc344bd19e1c4530244edeefa03fc924c65898ecff38b2159221640fbc3587b8 not found: ID does not exist"
Sep 05 02:52:02.763264 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.763243    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"]
Sep 05 02:52:02.767047 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:02.767026    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ca9bf-predictor-6bbb4bb6fb-bstjc"]
Sep 05 02:52:04.109341 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:04.109307    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" path="/var/lib/kubelet/pods/6a5b59a7-f1d9-4886-95ff-48f213b8ea08/volumes"
Sep 05 02:52:06.741468 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:06.741437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 02:52:06.741913 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:06.741889    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:52:07.688203 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:07.688166    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:52:16.742909 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:16.742868    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:52:17.688112 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:17.688074    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:52:26.742214 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:26.742172    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:52:27.688133 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:27.688089    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:52:36.742642 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:36.742593    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:52:37.689126 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:37.689095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 02:52:46.743064 ip-10-0-138-105 kubenswrapper[2582]: I0905 02:52:46.743030    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 03:01:14.110259 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:14.110225    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"]
Sep 05 03:01:14.112865 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:14.110455    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" containerID="cri-o://58ab0a7e91a86f1246aff6d509a3bad7db487842abe9f1ed13c900173d9445be" gracePeriod=30
Sep 05 03:01:14.112865 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:14.110492    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kube-rbac-proxy" containerID="cri-o://7b507c213df2021e03d9033e49ba9be4be735a4b3a3e7e4eb2e3c1cd40512d9a" gracePeriod=30
Sep 05 03:01:14.332659 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:14.332626    2582 generic.go:356] "Generic (PLEG): container finished" podID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerID="7b507c213df2021e03d9033e49ba9be4be735a4b3a3e7e4eb2e3c1cd40512d9a" exitCode=2
Sep 05 03:01:14.332824 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:14.332687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" event={"ID":"2eb1ea22-a856-4fd3-bfaa-22335d14af5d","Type":"ContainerDied","Data":"7b507c213df2021e03d9033e49ba9be4be735a4b3a3e7e4eb2e3c1cd40512d9a"}
Sep 05 03:01:16.736914 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:16.736869    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 05 03:01:16.742215 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:16.742186    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 03:01:17.347421 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.347390    2582 generic.go:356] "Generic (PLEG): container finished" podID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerID="58ab0a7e91a86f1246aff6d509a3bad7db487842abe9f1ed13c900173d9445be" exitCode=0
Sep 05 03:01:17.347556 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.347464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" event={"ID":"2eb1ea22-a856-4fd3-bfaa-22335d14af5d","Type":"ContainerDied","Data":"58ab0a7e91a86f1246aff6d509a3bad7db487842abe9f1ed13c900173d9445be"}
Sep 05 03:01:17.347556 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.347520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw" event={"ID":"2eb1ea22-a856-4fd3-bfaa-22335d14af5d","Type":"ContainerDied","Data":"1345ee8aa024a84b139bd56b0c94bc12b636ed77df5fffb2f5d9df8cca886f28"}
Sep 05 03:01:17.347556 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.347541    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1345ee8aa024a84b139bd56b0c94bc12b636ed77df5fffb2f5d9df8cca886f28"
Sep 05 03:01:17.355653 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.355633    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 03:01:17.447397 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.447314    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-proxy-tls\") pod \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") "
Sep 05 03:01:17.447397 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.447370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-rbac-proxy-sar-config\") pod \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") "
Sep 05 03:01:17.447590 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.447436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnjdr\" (UniqueName: \"kubernetes.io/projected/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-api-access-xnjdr\") pod \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\" (UID: \"2eb1ea22-a856-4fd3-bfaa-22335d14af5d\") "
Sep 05 03:01:17.447773 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.447743    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2eb1ea22-a856-4fd3-bfaa-22335d14af5d" (UID: "2eb1ea22-a856-4fd3-bfaa-22335d14af5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:01:17.449503 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.449476    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2eb1ea22-a856-4fd3-bfaa-22335d14af5d" (UID: "2eb1ea22-a856-4fd3-bfaa-22335d14af5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:01:17.449603 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.449498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-api-access-xnjdr" (OuterVolumeSpecName: "kube-api-access-xnjdr") pod "2eb1ea22-a856-4fd3-bfaa-22335d14af5d" (UID: "2eb1ea22-a856-4fd3-bfaa-22335d14af5d"). InnerVolumeSpecName "kube-api-access-xnjdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:01:17.548436 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.548398    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 03:01:17.548436 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.548429    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjdr\" (UniqueName: \"kubernetes.io/projected/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-kube-api-access-xnjdr\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 03:01:17.548436 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:17.548443    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eb1ea22-a856-4fd3-bfaa-22335d14af5d-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 03:01:18.352014 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:18.351981    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"
Sep 05 03:01:18.371708 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:18.371633    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"]
Sep 05 03:01:18.377062 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:18.377038    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a4a3e-predictor-df7cf9c54-h6kkw"]
Sep 05 03:01:20.109680 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:20.109648    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" path="/var/lib/kubelet/pods/2eb1ea22-a856-4fd3-bfaa-22335d14af5d/volumes"
Sep 05 03:01:40.354771 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:40.354737    2582 scope.go:117] "RemoveContainer" containerID="7b507c213df2021e03d9033e49ba9be4be735a4b3a3e7e4eb2e3c1cd40512d9a"
Sep 05 03:01:40.363116 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:01:40.363094    2582 scope.go:117] "RemoveContainer" containerID="58ab0a7e91a86f1246aff6d509a3bad7db487842abe9f1ed13c900173d9445be"
Sep 05 03:09:07.977206 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:07.977110    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"]
Sep 05 03:09:07.980056 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:07.977429    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container" containerID="cri-o://b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33" gracePeriod=30
Sep 05 03:09:07.980056 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:07.977444    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kube-rbac-proxy" containerID="cri-o://b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8" gracePeriod=30
Sep 05 03:09:08.508390 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:08.508351    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerID="b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8" exitCode=2
Sep 05 03:09:08.508574 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:08.508432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" event={"ID":"a7271d07-a22c-42cb-9e19-6587036c8eff","Type":"ContainerDied","Data":"b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8"}
Sep 05 03:09:11.139538 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.139510    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 03:09:11.268157 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.268062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-rbac-proxy-sar-config\") pod \"a7271d07-a22c-42cb-9e19-6587036c8eff\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") "
Sep 05 03:09:11.268334 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.268205    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls\") pod \"a7271d07-a22c-42cb-9e19-6587036c8eff\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") "
Sep 05 03:09:11.268334 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.268249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d96js\" (UniqueName: \"kubernetes.io/projected/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-api-access-d96js\") pod \"a7271d07-a22c-42cb-9e19-6587036c8eff\" (UID: \"a7271d07-a22c-42cb-9e19-6587036c8eff\") "
Sep 05 03:09:11.268516 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.268490    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7271d07-a22c-42cb-9e19-6587036c8eff" (UID: "a7271d07-a22c-42cb-9e19-6587036c8eff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:09:11.270354 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.270322    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7271d07-a22c-42cb-9e19-6587036c8eff" (UID: "a7271d07-a22c-42cb-9e19-6587036c8eff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:09:11.270466 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.270365    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-api-access-d96js" (OuterVolumeSpecName: "kube-api-access-d96js") pod "a7271d07-a22c-42cb-9e19-6587036c8eff" (UID: "a7271d07-a22c-42cb-9e19-6587036c8eff"). InnerVolumeSpecName "kube-api-access-d96js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:09:11.369774 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.369738    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96js\" (UniqueName: \"kubernetes.io/projected/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-api-access-d96js\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 03:09:11.369774 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.369766    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7271d07-a22c-42cb-9e19-6587036c8eff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 03:09:11.369774 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.369777    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7271d07-a22c-42cb-9e19-6587036c8eff-proxy-tls\") on node \"ip-10-0-138-105.ec2.internal\" DevicePath \"\""
Sep 05 03:09:11.522692 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.522598    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerID="b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33" exitCode=0
Sep 05 03:09:11.522692 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.522660    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"
Sep 05 03:09:11.522901 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.522688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" event={"ID":"a7271d07-a22c-42cb-9e19-6587036c8eff","Type":"ContainerDied","Data":"b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33"}
Sep 05 03:09:11.522901 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.522760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2" event={"ID":"a7271d07-a22c-42cb-9e19-6587036c8eff","Type":"ContainerDied","Data":"940c9ac085bf535690dacdf864cae8d1534f86ca674328f1dcbc44a9dbfe4b7d"}
Sep 05 03:09:11.522901 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.522798    2582 scope.go:117] "RemoveContainer" containerID="b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8"
Sep 05 03:09:11.531581 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.531563    2582 scope.go:117] "RemoveContainer" containerID="b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33"
Sep 05 03:09:11.539360 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.539340    2582 scope.go:117] "RemoveContainer" containerID="b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8"
Sep 05 03:09:11.539645 ip-10-0-138-105 kubenswrapper[2582]: E0905 03:09:11.539628    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8\": container with ID starting with b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8 not found: ID does not exist" containerID="b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8"
Sep 05 03:09:11.539681 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.539656    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8"} err="failed to get container status \"b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8\": rpc error: code = NotFound desc = could not find container \"b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8\": container with ID starting with b82b1e182490ced42a6bad2a41f60b5fb8c5698db6b9109abfba0c0c18cb8fd8 not found: ID does not exist"
Sep 05 03:09:11.539681 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.539672    2582 scope.go:117] "RemoveContainer" containerID="b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33"
Sep 05 03:09:11.539899 ip-10-0-138-105 kubenswrapper[2582]: E0905 03:09:11.539882    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33\": container with ID starting with b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33 not found: ID does not exist" containerID="b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33"
Sep 05 03:09:11.539936 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.539905    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33"} err="failed to get container status \"b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33\": rpc error: code = NotFound desc = could not find container \"b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33\": container with ID starting with b2a012218928e34e69e77f5788500bdc8c7ff5e2f353e8d3675031778b05bc33 not found: ID does not exist"
Sep 05 03:09:11.544430 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.544406    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"]
Sep 05 03:09:11.548588 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:11.548564    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3af6d-predictor-67876986cd-bjgx2"]
Sep 05 03:09:12.109158 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:12.109127    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" path="/var/lib/kubelet/pods/a7271d07-a22c-42cb-9e19-6587036c8eff/volumes"
Sep 05 03:09:40.180495 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:40.180467    2582 ???:1] "http: TLS handshake error from 10.0.138.105:51690: EOF"
Sep 05 03:09:40.186374 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:40.186352    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6rg87_676f3777-f8f1-4dcc-bf7b-6f95e34c07e2/global-pull-secret-syncer/0.log"
Sep 05 03:09:40.364903 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:40.364865    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xnglf_2b8e5e60-cb9f-4685-8abe-ce5d784ded6e/konnectivity-agent/0.log"
Sep 05 03:09:40.474388 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:40.474358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-105.ec2.internal_23947b3ee24ec342ed4ecf585dce8908/haproxy/0.log"
Sep 05 03:09:44.492135 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.492103    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/alertmanager/0.log"
Sep 05 03:09:44.516211 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.516183    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/config-reloader/0.log"
Sep 05 03:09:44.541497 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.541466    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/kube-rbac-proxy-web/0.log"
Sep 05 03:09:44.563195 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.563169    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/kube-rbac-proxy/0.log"
Sep 05 03:09:44.587727 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.587706    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/kube-rbac-proxy-metric/0.log"
Sep 05 03:09:44.613632 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.613610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/prom-label-proxy/0.log"
Sep 05 03:09:44.642150 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.642129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b324fcee-86cf-4848-a906-8fc42871cd5d/init-config-reloader/0.log"
Sep 05 03:09:44.728157 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.728134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-ljgvf_e2b5b2a6-f203-4cfb-981f-6d840d74173e/kube-state-metrics/0.log"
Sep 05 03:09:44.758293 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.758218    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-ljgvf_e2b5b2a6-f203-4cfb-981f-6d840d74173e/kube-rbac-proxy-main/0.log"
Sep 05 03:09:44.789715 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.789689    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-ljgvf_e2b5b2a6-f203-4cfb-981f-6d840d74173e/kube-rbac-proxy-self/0.log"
Sep 05 03:09:44.830376 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.830351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-669f5cbcd9-b762l_ee59885b-459e-46cc-9f75-f2a5ae98dfa6/metrics-server/0.log"
Sep 05 03:09:44.869239 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:44.869213    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-m8f6r_94cd9fea-764b-4d05-8cf8-d9d15b06e25f/monitoring-plugin/0.log"
Sep 05 03:09:45.121247 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.121218    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtpdm_553f165a-4f84-41e5-bc8f-5007cccfcd97/node-exporter/0.log"
Sep 05 03:09:45.143241 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.143216    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtpdm_553f165a-4f84-41e5-bc8f-5007cccfcd97/kube-rbac-proxy/0.log"
Sep 05 03:09:45.173112 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.173091    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vtpdm_553f165a-4f84-41e5-bc8f-5007cccfcd97/init-textfile/0.log"
Sep 05 03:09:45.208344 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.208319    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jdpgg_fc51dc11-4d1b-469c-b3b9-7487b5e2861d/kube-rbac-proxy-main/0.log"
Sep 05 03:09:45.233852 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.233829    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jdpgg_fc51dc11-4d1b-469c-b3b9-7487b5e2861d/kube-rbac-proxy-self/0.log"
Sep 05 03:09:45.261462 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.261436    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jdpgg_fc51dc11-4d1b-469c-b3b9-7487b5e2861d/openshift-state-metrics/0.log"
Sep 05 03:09:45.617468 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.617440    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d78687b65-dc25w_fa3e561e-aef3-4000-86cc-69d56b83e625/telemeter-client/0.log"
Sep 05 03:09:45.650752 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.650726    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d78687b65-dc25w_fa3e561e-aef3-4000-86cc-69d56b83e625/reload/0.log"
Sep 05 03:09:45.689828 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.689804    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d78687b65-dc25w_fa3e561e-aef3-4000-86cc-69d56b83e625/kube-rbac-proxy/0.log"
Sep 05 03:09:45.737514 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.737488    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-db4fbb5df-gwzz2_a8f9d653-c15b-4d51-bbde-2b05b30fef65/thanos-query/0.log"
Sep 05 03:09:45.772740 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.772713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-db4fbb5df-gwzz2_a8f9d653-c15b-4d51-bbde-2b05b30fef65/kube-rbac-proxy-web/0.log"
Sep 05 03:09:45.808253 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.808224    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-db4fbb5df-gwzz2_a8f9d653-c15b-4d51-bbde-2b05b30fef65/kube-rbac-proxy/0.log"
Sep 05 03:09:45.843386 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.843356    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-db4fbb5df-gwzz2_a8f9d653-c15b-4d51-bbde-2b05b30fef65/prom-label-proxy/0.log"
Sep 05 03:09:45.881628 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.881555    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-db4fbb5df-gwzz2_a8f9d653-c15b-4d51-bbde-2b05b30fef65/kube-rbac-proxy-rules/0.log"
Sep 05 03:09:45.912539 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:45.912512    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-db4fbb5df-gwzz2_a8f9d653-c15b-4d51-bbde-2b05b30fef65/kube-rbac-proxy-metrics/0.log"
Sep 05 03:09:47.121595 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:47.121565    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-w75fs_1887f748-d63e-440f-8097-21b65d22c227/networking-console-plugin/0.log"
Sep 05 03:09:48.054457 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.054427    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5788fcf757-hczxm_7ac71966-dc18-41de-8dcd-914f7002bf52/console/0.log"
Sep 05 03:09:48.250014 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.249976    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"]
Sep 05 03:09:48.250392 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250369    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container"
Sep 05 03:09:48.250392 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250381    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250403    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250413    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250434    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250444    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250452    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250461    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250472    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250480    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250489    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container"
Sep 05 03:09:48.250493 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container"
Sep 05 03:09:48.250868 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250585    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kserve-container"
Sep 05 03:09:48.250868 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250598    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kserve-container"
Sep 05 03:09:48.250868 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250609    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7271d07-a22c-42cb-9e19-6587036c8eff" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250868 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250618    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kserve-container"
Sep 05 03:09:48.250868 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250624    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb1ea22-a856-4fd3-bfaa-22335d14af5d" containerName="kube-rbac-proxy"
Sep 05 03:09:48.250868 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.250631    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5b59a7-f1d9-4886-95ff-48f213b8ea08" containerName="kube-rbac-proxy"
Sep 05 03:09:48.255021 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.254999    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.257782 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.257759    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zkkfn\"/\"default-dockercfg-5rzbt\""
Sep 05 03:09:48.257902 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.257778    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkkfn\"/\"kube-root-ca.crt\""
Sep 05 03:09:48.258618 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.258599    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkkfn\"/\"openshift-service-ca.crt\""
Sep 05 03:09:48.261408 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.261385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"]
Sep 05 03:09:48.290652 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.290616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-sys\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.290652 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.290651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-proc\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.290883 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.290672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977q8\" (UniqueName: \"kubernetes.io/projected/901eac77-e51e-442f-85f4-8440002b1a03-kube-api-access-977q8\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.290883 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.290748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-lib-modules\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.290883 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.290792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-podres\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391582 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-lib-modules\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391751 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-podres\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391751 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-sys\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391751 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-proc\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391751 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977q8\" (UniqueName: \"kubernetes.io/projected/901eac77-e51e-442f-85f4-8440002b1a03-kube-api-access-977q8\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391916 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-sys\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391916 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-proc\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391916 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-lib-modules\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.391916 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.391821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/901eac77-e51e-442f-85f4-8440002b1a03-podres\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.400351 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.400330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-977q8\" (UniqueName: \"kubernetes.io/projected/901eac77-e51e-442f-85f4-8440002b1a03-kube-api-access-977q8\") pod \"perf-node-gather-daemonset-8bwht\" (UID: \"901eac77-e51e-442f-85f4-8440002b1a03\") " pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.565218 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.565180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:48.839674 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:48.839639    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"]
Sep 05 03:09:48.842694 ip-10-0-138-105 kubenswrapper[2582]: W0905 03:09:48.842661    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod901eac77_e51e_442f_85f4_8440002b1a03.slice/crio-194c67bb44e9e105a6b4b44e1f5613ca3b01da56e5f6296df0aedc4e6b5693fd WatchSource:0}: Error finding container 194c67bb44e9e105a6b4b44e1f5613ca3b01da56e5f6296df0aedc4e6b5693fd: Status 404 returned error can't find the container with id 194c67bb44e9e105a6b4b44e1f5613ca3b01da56e5f6296df0aedc4e6b5693fd
Sep 05 03:09:49.646962 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.646934    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgvxp_f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5/dns/0.log"
Sep 05 03:09:49.667599 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.667574    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mgvxp_f2beaaa7-4e78-4f87-85ba-a22bb1adfcb5/kube-rbac-proxy/0.log"
Sep 05 03:09:49.690904 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.690871    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42lzd_2e7f06bf-d675-415e-8a76-1ec169531719/dns-node-resolver/0.log"
Sep 05 03:09:49.695030 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.695001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht" event={"ID":"901eac77-e51e-442f-85f4-8440002b1a03","Type":"ContainerStarted","Data":"81ad2c61a3566ec9e456c1c37b6b06d1b6b80e52b3cd9196a2b4ea846a82e072"}
Sep 05 03:09:49.695145 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.695055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht" event={"ID":"901eac77-e51e-442f-85f4-8440002b1a03","Type":"ContainerStarted","Data":"194c67bb44e9e105a6b4b44e1f5613ca3b01da56e5f6296df0aedc4e6b5693fd"}
Sep 05 03:09:49.695145 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.695083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:09:49.712602 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:49.712558    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht" podStartSLOduration=1.712544761 podStartE2EDuration="1.712544761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:09:49.711635908 +0000 UTC m=+3130.133141248" watchObservedRunningTime="2026-09-05 03:09:49.712544761 +0000 UTC m=+3130.134050099"
Sep 05 03:09:50.287457 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:50.287410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zhnn2_bd894870-c9ae-4904-a92f-6edba4702fbc/node-ca/0.log"
Sep 05 03:09:51.082985 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:51.082936    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-797dcfd87b-8dqvp_d3522c0a-02bd-42f1-8931-941e41b9151e/router/0.log"
Sep 05 03:09:51.427316 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:51.427225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-98vk8_5740d722-bab5-4ed8-9085-f5689745880b/serve-healthcheck-canary/0.log"
Sep 05 03:09:52.055012 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:52.054984    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvtld_0e549ee2-39c0-4387-b159-c7a58c93f6d3/kube-rbac-proxy/0.log"
Sep 05 03:09:52.078605 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:52.078569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvtld_0e549ee2-39c0-4387-b159-c7a58c93f6d3/exporter/0.log"
Sep 05 03:09:52.100436 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:52.100405    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvtld_0e549ee2-39c0-4387-b159-c7a58c93f6d3/extractor/0.log"
Sep 05 03:09:54.439021 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:54.438986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-tvwhg_0409ba3b-9adb-44a9-9545-f72625fd5a40/server/0.log"
Sep 05 03:09:54.529355 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:54.529312    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-zd6vp_c71f4124-85ed-4953-93af-19433318a583/manager/0.log"
Sep 05 03:09:54.559571 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:54.559543    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t26fm_56e01e05-076f-41a9-8464-4176ceeb4d7d/s3-init/0.log"
Sep 05 03:09:54.577075 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:54.577051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t26fm_56e01e05-076f-41a9-8464-4176ceeb4d7d/download-hf-model/0.log"
Sep 05 03:09:54.602813 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:54.602783    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-c287p_196c9abe-1bf7-492d-8794-fdf8dfa847fe/seaweedfs/0.log"
Sep 05 03:09:55.709932 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:09:55.709905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zkkfn/perf-node-gather-daemonset-8bwht"
Sep 05 03:10:00.545709 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.545676    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:10:00.569644 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.569612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/egress-router-binary-copy/0.log"
Sep 05 03:10:00.592034 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.591993    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/cni-plugins/0.log"
Sep 05 03:10:00.615283 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.615256    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/bond-cni-plugin/0.log"
Sep 05 03:10:00.637853 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.637824    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/routeoverride-cni/0.log"
Sep 05 03:10:00.660397 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.660372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/whereabouts-cni-bincopy/0.log"
Sep 05 03:10:00.683013 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:00.682960    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4hwq4_c5201b1d-866d-4e87-a351-5e7b24756371/whereabouts-cni/0.log"
Sep 05 03:10:01.001860 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:01.001830    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k48j2_a80efa97-46ac-46aa-94f8-187956a1d6a9/kube-multus/0.log"
Sep 05 03:10:01.073280 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:01.073255    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qkm56_9f623e00-8e5e-4eb1-8f53-3751e8474c52/network-metrics-daemon/0.log"
Sep 05 03:10:01.095242 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:01.095209    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qkm56_9f623e00-8e5e-4eb1-8f53-3751e8474c52/kube-rbac-proxy/0.log"
Sep 05 03:10:02.342124 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.342051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/ovn-controller/0.log"
Sep 05 03:10:02.362034 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.362006    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/ovn-acl-logging/0.log"
Sep 05 03:10:02.386245 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.386216    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/kube-rbac-proxy-node/0.log"
Sep 05 03:10:02.407801 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.407763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/kube-rbac-proxy-ovn-metrics/0.log"
Sep 05 03:10:02.434165 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.434131    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/northd/0.log"
Sep 05 03:10:02.457103 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.457077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/nbdb/0.log"
Sep 05 03:10:02.479446 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.479399    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/sbdb/0.log"
Sep 05 03:10:02.520815 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:02.520784    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njtb5_11061d9f-b38b-4629-a597-6071ab6c5908/ovnkube-controller/0.log"
Sep 05 03:10:03.566491 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:03.566460    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tvs2z_01e674ec-92bd-4c4d-9efc-78b1268bd8e8/network-check-target-container/0.log"
Sep 05 03:10:04.516014 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:04.515986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tlncs_93920abd-4a9c-45ac-805b-98dc5e2ff6ff/iptables-alerter/0.log"
Sep 05 03:10:05.248383 ip-10-0-138-105 kubenswrapper[2582]: I0905 03:10:05.248358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-clktc_db1b93ea-3186-4e0e-bde8-720c7ad651e7/tuned/0.log"