Jul 28 12:33:04.175875 ip-10-0-137-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 12:33:04.176013 ip-10-0-137-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 12:33:04.176099 ip-10-0-137-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 12:33:04.176372 ip-10-0-137-92 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 12:33:14.201830 ip-10-0-137-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 12:33:14.201846 ip-10-0-137-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2ab13c61dc8147878f270a7e9f03c323 --
Jul 28 12:34:50.104035 ip-10-0-137-92 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:34:50.573446 ip-10-0-137-92 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:34:50.573446 ip-10-0-137-92 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:34:50.573446 ip-10-0-137-92 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:34:50.573446 ip-10-0-137-92 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:34:50.573446 ip-10-0-137-92 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:34:50.574411 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.574326    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:34:50.578387 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578373    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:50.578387 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578387    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578391    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578395    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578398    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578401    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578404    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578406    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578409    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578411    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578414    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578416    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578419    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578421    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578424    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578426    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578429    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578431    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578435    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578439    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578442    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578444    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:50.578473 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578447    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578449    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578454    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578457    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578460    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578462    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578465    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578467    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578469    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578472    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578474    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578477    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578480    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578483    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578486    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578491    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578494    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:50.579021 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578504    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578507    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578510    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578513    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578516    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578518    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578521    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578524    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578526    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578529    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578531    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578534    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578537    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578540    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578543    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578545    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578548    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578551    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578554    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578556    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:50.579458 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578559    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578562    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578564    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578567    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578570    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578572    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578580    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578583    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578585    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578588    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578592    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578594    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578598    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578601    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578603    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578606    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578609    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578612    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578614    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578617    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:50.580001 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578621    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578624    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578627    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578630    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578635    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578639    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578642    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578646    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578649    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578652    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578655    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578657    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578660    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578662    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578665    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578667    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578670    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578673    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.578676    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:50.580483 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579463    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579471    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579475    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579478    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579482    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579484    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579487    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579493    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579496    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579499    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579502    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579504    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579507    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579510    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579512    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579515    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579517    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579521    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579525    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579528    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:50.580975 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579531    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579534    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579537    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579539    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579542    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579544    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579547    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579549    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579552    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579554    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579556    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579559    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579561    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579564    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579566    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579569    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579571    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579575    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579580    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:50.581519 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579583    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579587    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579589    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579591    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579594    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579596    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579599    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579602    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579604    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579607    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579609    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579612    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579614    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579616    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579619    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579621    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579624    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579626    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579629    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579631    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:50.582059 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579633    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579637    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579639    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579642    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579644    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579648    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579651    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579654    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579657    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579660    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579663    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579665    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579668    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579670    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579673    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579676    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579679    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579682    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579684    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:50.582566 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579687    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579689    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579692    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579694    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579697    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579700    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579702    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579704    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579707    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579709    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579711    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579715    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579718    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579720    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579723    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579726    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579728    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579731    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579733    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579736    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:50.583159 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.579738    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580135    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580141    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580144    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580147    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580150    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580153    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580156    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580159    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580161    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580165    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580167    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580170    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580173    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580175    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580178    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580181    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580183    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580186    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580188    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:50.583654 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580191    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580193    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580196    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580198    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580201    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580204    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580206    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580208    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580211    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580215    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580218    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580221    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580224    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580227    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580231    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580233    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580236    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580240    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580243    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:50.584166 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580245    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580250    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580257    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580260    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580263    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580266    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580268    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580271    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580273    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580276    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580278    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580281    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580286    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580289    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580292    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580294    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580297    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580299    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580302    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:50.584631 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580304    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580307    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580310    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580312    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580315    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580317    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580320    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580323    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580326    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580328    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580331    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580333    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580336    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580338    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580341    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580346    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580348    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580351    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580353    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:50.585123 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580355    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580358    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580361    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580363    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580366    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580368    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580370    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580373    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580375    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580378    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580380    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580382    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580385    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580387    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580390    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580392    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580394    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580397    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580399    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580402    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:50.585583 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580405    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580407    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.580410    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580487    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580494    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580500    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580505    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580509    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580516    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580521    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580525    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580529    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580532    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580535    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580539    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580542    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580545    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580548    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580551    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580554    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580557    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580562    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580565    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580568    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:34:50.586094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580571    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580574    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580578    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580581    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580584    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580588    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580594    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580597    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580600    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580603    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580606    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580611    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580613    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580617    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580620    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580623    2563 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580627    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580632    2563 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580635    2563 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580638    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580641    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580644    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580648    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580651    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580654    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:34:50.586667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580657    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580660    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580663    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580666    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580669    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580672    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580675    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580678    2563 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580681    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580684    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580687    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580691    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580694    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580699    2563 flags.go:64] FLAG: --help="false"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580702    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580705    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580708    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580711    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580714    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580718    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580721    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580724    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580727    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:34:50.587286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580731    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580734    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580737    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580741    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580744    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580747    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580750    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580752    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580755    2563 flags.go:64] FLAG: --lock-file=""
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580773    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580776    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580779    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580784    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580787    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580790    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580793    2563 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580796    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580800    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580802    2563 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580805    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580810    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580813    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580819    2563 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580822    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580825    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:34:50.587939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580828    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580831    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580834    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580837    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580840    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580847    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580850    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580855    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580858    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580861    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580867    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580871    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580874    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580877    2563 flags.go:64] FLAG: --port="10250"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580880    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580883    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b7e74df2766dd793"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580886    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580889    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580892    2563 flags.go:64] FLAG: --register-node="true"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580895    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580899    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580902    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580904    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580907    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580918    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:34:50.588541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580922    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580924    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580927    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580930    2563 flags.go:64] FLAG: --runonce="false"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580933    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580938    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580941    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580943    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580946    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580949    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580952    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580955    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580958    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580961    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580965    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580968    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580971    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580975    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580978    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580985    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580988    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.580996    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581001    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581004    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581006    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:34:50.589165 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581009    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581013    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581016    2563 flags.go:64] FLAG: --v="2"
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581020    2563 flags.go:64] FLAG: --version="false"
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581024    2563 flags.go:64] FLAG: --vmodule=""
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581028    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.581031    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581168    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581172    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581175    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581178    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581181    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581186    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581188    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581191    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581194    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581197    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581200    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581203    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581205    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581208    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:50.589779 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581211    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581214    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581217    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581220    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581223    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581226    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581230    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581233    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581235    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581238    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581241    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581243    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581246    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581250    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581254    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581257    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581259    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581262    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581265    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:50.590294 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581267    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581270    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581272    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581275    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581280    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581283    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581286    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581292    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581294    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581298    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581300    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581303    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581306    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581309    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581311    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581315    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581317    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581320    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581323    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581325    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:50.590746 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581328    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581331    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581334    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581336    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581339    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581342    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581344    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581347    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581350    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581352    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581354    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581357    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581359    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581362    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581365    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581367    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581372    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581375    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581377    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581380    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:50.591452 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581383    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581385    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581388    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581392    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581397    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581401    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581405    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581408    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581411    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581414    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581418    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581420    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581423    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581426    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581429    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581431    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581434    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581436    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581439    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:50.592215 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581441    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581444    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581447    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581449    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581452    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581455    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.581457    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.583441    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.591509    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:34:50.592688 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.591529    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:34:50.593098 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.592947    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:34:50.593128 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.593105    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:34:50.593184 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593175    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:50.593184 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593183    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593188    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593192    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593194    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593198    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593203    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593206    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593209    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593212    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593214    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593217    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593220    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593223    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593225    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593228    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593231    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593233    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593236    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593238    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:50.593234 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593241    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593244    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593247    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593250    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593252    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593255    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593257    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593259    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593270    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593273    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593275    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593278    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593281    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593284    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593286    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593289    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593292    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593294    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593297    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593299    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:50.593682 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593301    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593304    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593306    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593309    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593311    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593314    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593316    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593319    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593322    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593325    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593329    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593331    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593334    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593337    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593341    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593344    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593347    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593349    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593352    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593354    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:50.594183 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593357    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593359    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593362    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593364    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593367    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593369    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593372    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593374    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593377    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593379    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593382    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593384    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593387    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593389    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593392    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593394    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593397    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593399    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593402    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593404    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:50.594665 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593407    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593409    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593412    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593415    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593417    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593420    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593423    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593426    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593428    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593431    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593433    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593436    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593439    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593441    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593445    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593448    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593450    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593453    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:50.595250 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593456    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.593461    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593605    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593610    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593613    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593617    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593619    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593622    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593625    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593628    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593631    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593635    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593639    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593642    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593647    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:34:50.595686 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593649    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593652    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593654    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593657    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593660    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593662    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593665    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593668    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593670    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593673    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593675    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593678    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593681    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593683    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593686    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593689    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593692    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593694    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593697    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593699    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:34:50.596063 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593702    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593704    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593707    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593710    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593713    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593715    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593719    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593723    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593726    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593729    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593732    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593735    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593738    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593741    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593743    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593746    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593749    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593751    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593753    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:34:50.596547 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593777    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593782    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593786    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593788    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593791    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593794    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593796    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593799    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593801    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593804    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593806    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593809    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593811    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593816    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593819    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593822    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593825    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593827    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593829    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:34:50.597042 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593832    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593834    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593837    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593839    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593842    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593844    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593850    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593853    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593856    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593859    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593861    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593864    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593867    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593869    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593871    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593874    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593876    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593879    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593881    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:34:50.597523 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593883    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593886    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593888    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593891    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593893    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593895    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593898    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:50.593900    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.593905    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.594028    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.596003    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:34:50.597993 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.597904    2563 server.go:1019] "Starting client certificate rotation"
Jul 28 12:34:50.598261 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.598006    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:34:50.598261 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.598038    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:34:50.601298 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.601258    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:34:50.603231 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.603214    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:34:50.612078 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.612062    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:34:50.616396 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.616378    2563 log.go:25] "Validated CRI v1 image API"
Jul 28 12:34:50.617291 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.617276    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:34:50.624869 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.624844    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9326856b-5696-4926-961d-6420939f0c0c:/dev/nvme0n1p4 db981678-45c1-4f6d-a13a-227d7ede1e20:/dev/nvme0n1p3]
Jul 28 12:34:50.624942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.624869    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:34:50.630387 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.630271    2563 manager.go:216] Machine: {Timestamp:2026-07-28 12:34:50.628330995 +0000 UTC m=+0.383744065 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3068223 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec236be057c9c7f5498a1ca213a6d017 SystemUUID:ec236be0-57c9-c7f5-498a-1ca213a6d017 BootID:2ab13c61-dc81-4787-8f27-0a7e9f03c323 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:64:8e:24:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:64:8e:24:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:a4:aa:44:fb:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:34:50.630387 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.630380    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:34:50.630525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.630462    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:34:50.631216 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.631200    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:34:50.633125 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.633097    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:34:50.633257 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.633127    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:34:50.633844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.633832    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:34:50.633873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.633848    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:34:50.633873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.633855    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:34:50.634569 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.634560    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:34:50.635478 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.635468    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:34:50.637123 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637111    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:34:50.637270 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637261    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:34:50.637304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637294    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:34:50.637337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637314    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:34:50.637337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637331    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:34:50.637390 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637345    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:34:50.637390 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637358    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:34:50.637867 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637851    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:34:50.637914 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.637888    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:34:50.641356 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.641330    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:34:50.643298 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.643284    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:34:50.643349 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.643315    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:34:50.645137 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645120    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:34:50.645137 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645138    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645144    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645151    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645159    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645164    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645171    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645176    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645183    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645188    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645196    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:34:50.645233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.645201    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:34:50.646087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.646078    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:34:50.646087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.646087    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:34:50.649299 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.649286    2563 server.go:1268] "Started kubelet"
Jul 28 12:34:50.649419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.649361    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:34:50.649419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.649401    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:34:50.649513 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.649437    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:34:50.649641 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.649621    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:34:50.649703 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.649630    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:34:50.649779 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.649742    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:34:50.650113 ip-10-0-137-92 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:34:50.651355 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.651334    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:34:50.651465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.651376    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:34:50.655956 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.655936    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:34:50.656109 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.654969    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-92.ec2.internal.18c674390b398d48  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-92.ec2.internal,UID:ip-10-0-137-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-92.ec2.internal,},FirstTimestamp:2026-07-28 12:34:50.649259336 +0000 UTC m=+0.404672404,LastTimestamp:2026-07-28 12:34:50.649259336 +0000 UTC m=+0.404672404,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-92.ec2.internal,}"
Jul 28 12:34:50.660703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.660686    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:34:50.660703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.660698    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:34:50.660868 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.660754    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:50.662142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.661449    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:34:50.662142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.661452    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:34:50.662142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.661676    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:34:50.662142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.661881    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:34:50.662142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.661892    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:34:50.662672 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.662656    2563 factory.go:55] Registering systemd factory
Jul 28 12:34:50.662751 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.662690    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:34:50.662972 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.662960    2563 factory.go:153] Registering CRI-O factory
Jul 28 12:34:50.663075 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.663064    2563 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:34:50.663201 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.663184    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:34:50.663837 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.663218    2563 factory.go:103] Registering Raw factory
Jul 28 12:34:50.663837 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.663233    2563 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:34:50.663837 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.663266    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:34:50.663837 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.663276    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:34:50.665036 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.665020    2563 manager.go:349] Starting recovery of all containers
Jul 28 12:34:50.675418 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.675300    2563 manager.go:354] Recovery completed
Jul 28 12:34:50.680290 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.680277    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.682094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682077    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.682164 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682107    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.682164 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682119    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.682608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682591    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:34:50.682608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682606    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:34:50.682781 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682619    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.682781 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.682624    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:34:50.683297 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.683283    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.683368 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.683316    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.683368 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.683331    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.684406 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.684394    2563 policy_none.go:49] "None policy: Start"
Jul 28 12:34:50.684465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.684410    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:34:50.684465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.684420    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:34:50.685222 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.685211    2563 policy_none.go:47] "Start"
Jul 28 12:34:50.686500 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.686485    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pgqwd"
Jul 28 12:34:50.691950 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.691880    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-92.ec2.internal.18c674390d2e9427  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-92.ec2.internal,UID:ip-10-0-137-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-92.ec2.internal,},FirstTimestamp:2026-07-28 12:34:50.682094631 +0000 UTC m=+0.437507699,LastTimestamp:2026-07-28 12:34:50.682094631 +0000 UTC m=+0.437507699,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-92.ec2.internal,}"
Jul 28 12:34:50.693945 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.693930    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pgqwd"
Jul 28 12:34:50.719506 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.719434    2563 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:34:50.719506 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.719475    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:34:50.719506 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.719488    2563 server.go:87] "Starting device plugin registration server"
Jul 28 12:34:50.719814 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.719799    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:34:50.719888 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.719817    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:34:50.719939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.719902    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:34:50.720024 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.719996    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:34:50.720024 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.720017    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:34:50.720604 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.720586    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:34:50.720685 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.720634    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:50.746132 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.746109    2563 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2596 into the system container \"/system.slice\": <nil>"
Jul 28 12:34:50.799958 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.799930    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:34:50.801412 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.801392    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:34:50.801482 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.801439    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:34:50.801482 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.801464    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:34:50.801567 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.801500    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:34:50.804347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.804330    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:50.820863 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.820847    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.821629 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.821616    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.821681 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.821646    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.821681 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.821658    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.821681 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.821679    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.832425 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.832407    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.832474 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.832430    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-92.ec2.internal\": node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:50.857886 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.857862    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:50.901608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.901580    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"]
Jul 28 12:34:50.901689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.901676    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.902620 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.902604    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.902703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.902638    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.902703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.902652    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.903957 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.903942    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.904098 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.904142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904113    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.904665 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904649    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.904746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904676    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.904746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904688    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.904746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904706    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.904746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904725    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.904746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.904735    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.905790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.905775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.905866 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.905809    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:34:50.906489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.906475    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:34:50.906562 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.906503    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:34:50.906562 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.906517    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:34:50.930490 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.930467    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-92.ec2.internal\" not found" node="ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.934947 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.934931    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-92.ec2.internal\" not found" node="ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.958368 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:50.958337    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:50.963717 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.963700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81197a6b43549ebbb8245f4ccb902654-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"81197a6b43549ebbb8245f4ccb902654\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.963815 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.963732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81197a6b43549ebbb8245f4ccb902654-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"81197a6b43549ebbb8245f4ccb902654\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:50.963815 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:50.963777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e0ede8a7a9a486a43a28960227e9b07d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-92.ec2.internal\" (UID: \"e0ede8a7a9a486a43a28960227e9b07d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.059093 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.059006    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:51.064417 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.064391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81197a6b43549ebbb8245f4ccb902654-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"81197a6b43549ebbb8245f4ccb902654\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.064467 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.064427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81197a6b43549ebbb8245f4ccb902654-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"81197a6b43549ebbb8245f4ccb902654\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.064467 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.064448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e0ede8a7a9a486a43a28960227e9b07d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-92.ec2.internal\" (UID: \"e0ede8a7a9a486a43a28960227e9b07d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.064539 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.064490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81197a6b43549ebbb8245f4ccb902654-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"81197a6b43549ebbb8245f4ccb902654\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.064539 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.064501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e0ede8a7a9a486a43a28960227e9b07d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-92.ec2.internal\" (UID: \"e0ede8a7a9a486a43a28960227e9b07d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.064539 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.064509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81197a6b43549ebbb8245f4ccb902654-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal\" (UID: \"81197a6b43549ebbb8245f4ccb902654\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.159851 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.159815    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:51.233312 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.233280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.237995 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.237980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.260638 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.260614    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:51.361105 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.361019    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:51.461475 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.461430    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:51.561979 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.561945    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-92.ec2.internal\" not found"
Jul 28 12:34:51.590162 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.590142    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:51.596477 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.596459    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:34:51.596573 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.596558    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:34:51.596624 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.596604    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:34:51.637835 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.637749    2563 apiserver.go:52] "Watching apiserver"
Jul 28 12:34:51.656541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.656519    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:34:51.656878 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.656860    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4vz9w","openshift-image-registry/node-ca-6wrq8","openshift-network-operator/iptables-alerter-57pb8","openshift-ovn-kubernetes/ovnkube-node-k7tvz","kube-system/konnectivity-agent-cptn6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv","openshift-cluster-node-tuning-operator/tuned-mw6vl","openshift-multus/multus-additional-cni-plugins-9gc9r","openshift-multus/multus-z5bdt","openshift-multus/network-metrics-daemon-dqk86","openshift-network-diagnostics/network-check-target-5tsz9","kube-system/global-pull-secret-syncer-dh4h9"]
Jul 28 12:34:51.658148 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.658134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.661211 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.659597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.661211 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.661114    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.661334 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.661269    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:34:51.661828 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.661805    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.662003 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.661983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.662079 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.662054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.662245 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.662227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gpzk4\""
Jul 28 12:34:51.663679 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.663656    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:34:51.663829 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.663698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.663829 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.663712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.663829 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.663728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jk5jd\""
Jul 28 12:34:51.664506 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.664487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.665166 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.664959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.665166 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.665065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pfwfj\""
Jul 28 12:34:51.665417 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.665403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.665569 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.665551    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:34:51.666038 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.666023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.667136 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.667225 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-systemd\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667225 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667225 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71572878-eed8-4109-913e-eee642504dba-hosts-file\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.667366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-ovn\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-cni-netd\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-host\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.667366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/343e89fd-3c76-4dcd-8154-d28f1ba2daec-iptables-alerter-script\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.667537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-node-log\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-log-socket\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-env-overrides\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmfb\" (UniqueName: \"kubernetes.io/projected/343e89fd-3c76-4dcd-8154-d28f1ba2daec-kube-api-access-ggmfb\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.667537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-kubelet\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-etc-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5l5\" (UniqueName: \"kubernetes.io/projected/71572878-eed8-4109-913e-eee642504dba-kube-api-access-4j5l5\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-run-netns\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-var-lib-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-ovnkube-script-lib\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-serviceca\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-systemd-units\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667699 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-slash\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-cni-bin\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcwl\" (UniqueName: \"kubernetes.io/projected/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-kube-api-access-dxcwl\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7b35311-e67c-403b-afd8-7755a95f64a5-ovn-node-metrics-cert\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343e89fd-3c76-4dcd-8154-d28f1ba2daec-host-slash\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-ovnkube-config\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzc8d\" (UniqueName: \"kubernetes.io/projected/b7b35311-e67c-403b-afd8-7755a95f64a5-kube-api-access-qzc8d\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.667970 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.667873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71572878-eed8-4109-913e-eee642504dba-tmp-dir\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.668275 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.668275 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:34:51.668275 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.668463 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.668463 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668286    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:34:51.668463 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668346    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7b9w6\""
Jul 28 12:34:51.668589 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668489    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:34:51.668626 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-klt45\""
Jul 28 12:34:51.668626 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:34:51.668626 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668621    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:34:51.668750 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.668648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:34:51.669399 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.669379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hm8wr\""
Jul 28 12:34:51.669475 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.669397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:34:51.669475 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.669457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.669567 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.669502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.670077 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.670060    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.670558 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.670541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.671213 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.671196    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.671213 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.671205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsxjm\""
Jul 28 12:34:51.671577 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.671562    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.671672 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.671657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:51.671753 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.671735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:34:51.672892 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.672874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:51.672969 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.672944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:34:51.673035 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:34:51.673083 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:34:51.673299 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:34:51.673371 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:34:51.673442 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673423    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:34:51.673660 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673645    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:34:51.673719 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673692    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6xxws\""
Jul 28 12:34:51.673719 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.673706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jtwkl\""
Jul 28 12:34:51.674387 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.674370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.674463 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.674430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:34:51.675978 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.675961    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:34:51.681433 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.681419    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:34:51.681501 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.681488    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"
Jul 28 12:34:51.681615 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.681601    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal"]
Jul 28 12:34:51.691124 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.691107    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal"]
Jul 28 12:34:51.691192 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.691110    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:34:51.699134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.699107    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:29:52 +0000 UTC" deadline="2028-01-19 19:32:16.366209968 +0000 UTC"
Jul 28 12:34:51.699198 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.699134    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12966h57m24.667077946s"
Jul 28 12:34:51.705951 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.705935    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jc4bx"
Jul 28 12:34:51.714345 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.714327    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jc4bx"
Jul 28 12:34:51.738697 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.738678    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:51.763016 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.762993    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:34:51.768103 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-modprobe-d\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.768219 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8db7ce2-f506-453a-8d17-36c233734369-multus-daemon-config\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768219 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-os-release\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768219 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-etc-kubernetes\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768219 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysctl-conf\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.768219 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-var-lib-kubelet\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-k8s-cni-cncf-io\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71572878-eed8-4109-913e-eee642504dba-hosts-file\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-ovn\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-cni-netd\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/71572878-eed8-4109-913e-eee642504dba-hosts-file\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-host\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-ovn\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-host\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-cni-netd\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768419 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d62d21db-38dd-451a-967d-91a67fa38c39-konnectivity-ca\") pod \"konnectivity-agent-cptn6\" (UID: \"d62d21db-38dd-451a-967d-91a67fa38c39\") " pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-system-cni-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-netns\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-node-log\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-log-socket\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cnibin\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-registration-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-kubernetes\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-node-log\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-log-socket\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-conf-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-kubelet\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-etc-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-kubelet\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.768787 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-system-cni-dir\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-etc-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrfb\" (UniqueName: \"kubernetes.io/projected/f8db7ce2-f506-453a-8d17-36c233734369-kube-api-access-jbrfb\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-var-lib-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-serviceca\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-os-release\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30fd4493-5528-42f8-b77e-a73c763a2116-dbus\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-var-lib-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-socket-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.768969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-cnibin\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-cni-multus\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-systemd-units\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-systemd-units\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-cni-bin\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcwl\" (UniqueName: \"kubernetes.io/projected/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-kube-api-access-dxcwl\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-cni-bin\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.769465 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2prl\" (UniqueName: \"kubernetes.io/projected/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-kube-api-access-d2prl\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-sys\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fd7j\" (UniqueName: \"kubernetes.io/projected/bb855a11-5572-4be0-a3ba-a0830feedecf-kube-api-access-4fd7j\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7b35311-e67c-403b-afd8-7755a95f64a5-ovn-node-metrics-cert\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30fd4493-5528-42f8-b77e-a73c763a2116-kubelet-config\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdfs\" (UniqueName: \"kubernetes.io/projected/5b10be79-9adb-469e-8d70-776c649328f9-kube-api-access-9xdfs\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysctl-d\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343e89fd-3c76-4dcd-8154-d28f1ba2daec-host-slash\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-ovnkube-config\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343e89fd-3c76-4dcd-8154-d28f1ba2daec-host-slash\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769748    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:34:51.770207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.769885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-serviceca\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-ovnkube-config\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzc8d\" (UniqueName: \"kubernetes.io/projected/b7b35311-e67c-403b-afd8-7755a95f64a5-kube-api-access-qzc8d\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71572878-eed8-4109-913e-eee642504dba-tmp-dir\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-sys-fs\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-systemd\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-kubelet\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-systemd\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-systemd\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-device-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-run\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71572878-eed8-4109-913e-eee642504dba-tmp-dir\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-lib-modules\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.770785 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8db7ce2-f506-453a-8d17-36c233734369-cni-binary-copy\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-hostroot\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-tuned\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/343e89fd-3c76-4dcd-8154-d28f1ba2daec-iptables-alerter-script\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.770980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-env-overrides\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d62d21db-38dd-451a-967d-91a67fa38c39-agent-certs\") pod \"konnectivity-agent-cptn6\" (UID: \"d62d21db-38dd-451a-967d-91a67fa38c39\") " pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysconfig\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-cni-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-socket-dir-parent\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-multus-certs\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmfb\" (UniqueName: \"kubernetes.io/projected/343e89fd-3c76-4dcd-8154-d28f1ba2daec-kube-api-access-ggmfb\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5l5\" (UniqueName: \"kubernetes.io/projected/71572878-eed8-4109-913e-eee642504dba-kube-api-access-4j5l5\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-host\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-cni-bin\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-run-netns\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-ovnkube-script-lib\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb855a11-5572-4be0-a3ba-a0830feedecf-tmp\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2tg\" (UniqueName: \"kubernetes.io/projected/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-kube-api-access-lt2tg\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-slash\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-run-netns\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-ovnkube-script-lib\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.771922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.771905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-host-slash\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.772347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.772032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b7b35311-e67c-403b-afd8-7755a95f64a5-env-overrides\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.772347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.772295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b35311-e67c-403b-afd8-7755a95f64a5-run-openvswitch\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.772347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.772302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/343e89fd-3c76-4dcd-8154-d28f1ba2daec-iptables-alerter-script\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.773586 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.773559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7b35311-e67c-403b-afd8-7755a95f64a5-ovn-node-metrics-cert\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.778170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.778149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzc8d\" (UniqueName: \"kubernetes.io/projected/b7b35311-e67c-403b-afd8-7755a95f64a5-kube-api-access-qzc8d\") pod \"ovnkube-node-k7tvz\" (UID: \"b7b35311-e67c-403b-afd8-7755a95f64a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:51.778313 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.778297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcwl\" (UniqueName: \"kubernetes.io/projected/5cd0dcaa-46fa-4da7-a97e-c8737a1bf049-kube-api-access-dxcwl\") pod \"node-ca-6wrq8\" (UID: \"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049\") " pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:51.779888 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.779869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmfb\" (UniqueName: \"kubernetes.io/projected/343e89fd-3c76-4dcd-8154-d28f1ba2daec-kube-api-access-ggmfb\") pod \"iptables-alerter-57pb8\" (UID: \"343e89fd-3c76-4dcd-8154-d28f1ba2daec\") " pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:51.779967 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.779868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5l5\" (UniqueName: \"kubernetes.io/projected/71572878-eed8-4109-913e-eee642504dba-kube-api-access-4j5l5\") pod \"node-resolver-4vz9w\" (UID: \"71572878-eed8-4109-913e-eee642504dba\") " pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.872683 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-sys-fs\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-systemd\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-kubelet\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-kubelet\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-systemd\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-device-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.872844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-run\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-lib-modules\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-sys-fs\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-device-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8db7ce2-f506-453a-8d17-36c233734369-cni-binary-copy\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-hostroot\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-tuned\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.872997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-lib-modules\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-run\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-hostroot\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d62d21db-38dd-451a-967d-91a67fa38c39-agent-certs\") pod \"konnectivity-agent-cptn6\" (UID: \"d62d21db-38dd-451a-967d-91a67fa38c39\") " pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysconfig\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-cni-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-socket-dir-parent\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-multus-certs\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-host\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.873236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-cni-bin\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb855a11-5572-4be0-a3ba-a0830feedecf-tmp\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2tg\" (UniqueName: \"kubernetes.io/projected/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-kube-api-access-lt2tg\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-socket-dir-parent\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysconfig\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-modprobe-d\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-cni-bin\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-modprobe-d\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-multus-certs\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8db7ce2-f506-453a-8d17-36c233734369-multus-daemon-config\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8db7ce2-f506-453a-8d17-36c233734369-cni-binary-copy\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-os-release\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-host\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-etc-kubernetes\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysctl-conf\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-os-release\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-var-lib-kubelet\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-k8s-cni-cncf-io\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-etc-kubernetes\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d62d21db-38dd-451a-967d-91a67fa38c39-konnectivity-ca\") pod \"konnectivity-agent-cptn6\" (UID: \"d62d21db-38dd-451a-967d-91a67fa38c39\") " pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-var-lib-kubelet\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-system-cni-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-cni-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-netns\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cnibin\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-k8s-cni-cncf-io\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-registration-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysctl-conf\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-kubernetes\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-conf-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cnibin\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.874974 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-run-netns\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-system-cni-dir\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-multus-conf-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-registration-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-system-cni-dir\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrfb\" (UniqueName: \"kubernetes.io/projected/f8db7ce2-f506-453a-8d17-36c233734369-kube-api-access-jbrfb\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-system-cni-dir\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-os-release\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.873987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30fd4493-5528-42f8-b77e-a73c763a2116-dbus\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-socket-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-cnibin\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-cni-multus\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2prl\" (UniqueName: \"kubernetes.io/projected/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-kube-api-access-d2prl\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-sys\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fd7j\" (UniqueName: \"kubernetes.io/projected/bb855a11-5572-4be0-a3ba-a0830feedecf-kube-api-access-4fd7j\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.875555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8db7ce2-f506-453a-8d17-36c233734369-multus-daemon-config\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-cnibin\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30fd4493-5528-42f8-b77e-a73c763a2116-kubelet-config\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-os-release\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdfs\" (UniqueName: \"kubernetes.io/projected/5b10be79-9adb-469e-8d70-776c649328f9-kube-api-access-9xdfs\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysctl-d\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-sys\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b10be79-9adb-469e-8d70-776c649328f9-socket-dir\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8db7ce2-f506-453a-8d17-36c233734369-host-var-lib-cni-multus\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-kubernetes\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d62d21db-38dd-451a-967d-91a67fa38c39-konnectivity-ca\") pod \"konnectivity-agent-cptn6\" (UID: \"d62d21db-38dd-451a-967d-91a67fa38c39\") " pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30fd4493-5528-42f8-b77e-a73c763a2116-dbus\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.874596    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:51.876308 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.874672    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:52.37465207 +0000 UTC m=+2.130065143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.874720    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.874818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:52.374799533 +0000 UTC m=+2.130212607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-sysctl-d\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.874884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30fd4493-5528-42f8-b77e-a73c763a2116-kubelet-config\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.875032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.875653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb855a11-5572-4be0-a3ba-a0830feedecf-etc-tuned\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.875679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb855a11-5572-4be0-a3ba-a0830feedecf-tmp\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.876808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.876682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d62d21db-38dd-451a-967d-91a67fa38c39-agent-certs\") pod \"konnectivity-agent-cptn6\" (UID: \"d62d21db-38dd-451a-967d-91a67fa38c39\") " pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:51.881719 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:51.881685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81197a6b43549ebbb8245f4ccb902654.slice/crio-3d648727fa321b9d14ea7093989b02dc4e509f3784a7201955e2f24fc41da097 WatchSource:0}: Error finding container 3d648727fa321b9d14ea7093989b02dc4e509f3784a7201955e2f24fc41da097: Status 404 returned error can't find the container with id 3d648727fa321b9d14ea7093989b02dc4e509f3784a7201955e2f24fc41da097
Jul 28 12:34:51.882704 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:51.882688    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ede8a7a9a486a43a28960227e9b07d.slice/crio-48870ea590b4bb357708eceafdef20d7a16621182006e4f4c98550bd48e354d9 WatchSource:0}: Error finding container 48870ea590b4bb357708eceafdef20d7a16621182006e4f4c98550bd48e354d9: Status 404 returned error can't find the container with id 48870ea590b4bb357708eceafdef20d7a16621182006e4f4c98550bd48e354d9
Jul 28 12:34:51.885865 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.885847    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:51.885933 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.885868    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:51.885933 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.885877    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:51.886007 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:51.885948    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:52.385931723 +0000 UTC m=+2.141344784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:51.888010 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.887954    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:34:51.888428 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.888409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2prl\" (UniqueName: \"kubernetes.io/projected/e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6-kube-api-access-d2prl\") pod \"multus-additional-cni-plugins-9gc9r\" (UID: \"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6\") " pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:51.888602 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.888583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2tg\" (UniqueName: \"kubernetes.io/projected/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-kube-api-access-lt2tg\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:51.888682 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.888664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrfb\" (UniqueName: \"kubernetes.io/projected/f8db7ce2-f506-453a-8d17-36c233734369-kube-api-access-jbrfb\") pod \"multus-z5bdt\" (UID: \"f8db7ce2-f506-453a-8d17-36c233734369\") " pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:51.889660 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.889641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fd7j\" (UniqueName: \"kubernetes.io/projected/bb855a11-5572-4be0-a3ba-a0830feedecf-kube-api-access-4fd7j\") pod \"tuned-mw6vl\" (UID: \"bb855a11-5572-4be0-a3ba-a0830feedecf\") " pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:51.889908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.889893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdfs\" (UniqueName: \"kubernetes.io/projected/5b10be79-9adb-469e-8d70-776c649328f9-kube-api-access-9xdfs\") pod \"aws-ebs-csi-driver-node-c2bfv\" (UID: \"5b10be79-9adb-469e-8d70-776c649328f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:51.981068 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.981031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vz9w"
Jul 28 12:34:51.991095 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:51.991069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71572878_eed8_4109_913e_eee642504dba.slice/crio-9361fe71e1418be6b38375ee05ce3c269b823e9da6dd43061e55175f509072e5 WatchSource:0}: Error finding container 9361fe71e1418be6b38375ee05ce3c269b823e9da6dd43061e55175f509072e5: Status 404 returned error can't find the container with id 9361fe71e1418be6b38375ee05ce3c269b823e9da6dd43061e55175f509072e5
Jul 28 12:34:51.995233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:51.995215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wrq8"
Jul 28 12:34:52.002159 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.002141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57pb8"
Jul 28 12:34:52.003429 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.003406    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd0dcaa_46fa_4da7_a97e_c8737a1bf049.slice/crio-4f4469df5c1230db2b0b9db2f93edc7d134fc4d636c57df98a6c7b911f35bc47 WatchSource:0}: Error finding container 4f4469df5c1230db2b0b9db2f93edc7d134fc4d636c57df98a6c7b911f35bc47: Status 404 returned error can't find the container with id 4f4469df5c1230db2b0b9db2f93edc7d134fc4d636c57df98a6c7b911f35bc47
Jul 28 12:34:52.010842 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.010822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343e89fd_3c76_4dcd_8154_d28f1ba2daec.slice/crio-ebf7e3cf4b0cf1f8383bb4608672281c5db5ca9df64d11b9e9e2a1993a11e2d3 WatchSource:0}: Error finding container ebf7e3cf4b0cf1f8383bb4608672281c5db5ca9df64d11b9e9e2a1993a11e2d3: Status 404 returned error can't find the container with id ebf7e3cf4b0cf1f8383bb4608672281c5db5ca9df64d11b9e9e2a1993a11e2d3
Jul 28 12:34:52.029968 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.029943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:34:52.038260 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.038232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b35311_e67c_403b_afd8_7755a95f64a5.slice/crio-d5e85a1f31926dcdfb97a85251edfad21fbb7435175500d002824a397deadbf7 WatchSource:0}: Error finding container d5e85a1f31926dcdfb97a85251edfad21fbb7435175500d002824a397deadbf7: Status 404 returned error can't find the container with id d5e85a1f31926dcdfb97a85251edfad21fbb7435175500d002824a397deadbf7
Jul 28 12:34:52.045486 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.045466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:34:52.053467 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.053439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62d21db_38dd_451a_967d_91a67fa38c39.slice/crio-07481522c3e899cc4058ccf2baa23f03d91eba74dd3ba997cf61b52dfdb7fc03 WatchSource:0}: Error finding container 07481522c3e899cc4058ccf2baa23f03d91eba74dd3ba997cf61b52dfdb7fc03: Status 404 returned error can't find the container with id 07481522c3e899cc4058ccf2baa23f03d91eba74dd3ba997cf61b52dfdb7fc03
Jul 28 12:34:52.068068 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.068051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv"
Jul 28 12:34:52.075518 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.075498    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b10be79_9adb_469e_8d70_776c649328f9.slice/crio-debf2fa5820088da50b1846d65ce0cafdf63af09833ca6952d24dfc030f7bd5c WatchSource:0}: Error finding container debf2fa5820088da50b1846d65ce0cafdf63af09833ca6952d24dfc030f7bd5c: Status 404 returned error can't find the container with id debf2fa5820088da50b1846d65ce0cafdf63af09833ca6952d24dfc030f7bd5c
Jul 28 12:34:52.090771 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.090744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mw6vl"
Jul 28 12:34:52.098818 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.098797    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb855a11_5572_4be0_a3ba_a0830feedecf.slice/crio-e46ba3517ac0f4d5ad30a35bab043afb0c21a32936a96cf7f8e25da9bcd9cd26 WatchSource:0}: Error finding container e46ba3517ac0f4d5ad30a35bab043afb0c21a32936a96cf7f8e25da9bcd9cd26: Status 404 returned error can't find the container with id e46ba3517ac0f4d5ad30a35bab043afb0c21a32936a96cf7f8e25da9bcd9cd26
Jul 28 12:34:52.107711 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.107695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gc9r"
Jul 28 12:34:52.112486 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.112468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5bdt"
Jul 28 12:34:52.116677 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.116657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4eebd04_e2b5_449b_843f_dd9f5dd1f9c6.slice/crio-0c0d654192825f6bd4304f8d0d7c412307f67624934ad6a9f24610934376acf4 WatchSource:0}: Error finding container 0c0d654192825f6bd4304f8d0d7c412307f67624934ad6a9f24610934376acf4: Status 404 returned error can't find the container with id 0c0d654192825f6bd4304f8d0d7c412307f67624934ad6a9f24610934376acf4
Jul 28 12:34:52.121545 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:34:52.121524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8db7ce2_f506_453a_8d17_36c233734369.slice/crio-b631087b2a3fe6f138d3eca3a1e9591401d5999800ed41d4d16a15abdc9f1171 WatchSource:0}: Error finding container b631087b2a3fe6f138d3eca3a1e9591401d5999800ed41d4d16a15abdc9f1171: Status 404 returned error can't find the container with id b631087b2a3fe6f138d3eca3a1e9591401d5999800ed41d4d16a15abdc9f1171
Jul 28 12:34:52.183795 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.183710    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:52.377707 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.377670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:52.377873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.377723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:52.377873 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.377855    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:52.377873 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.377861    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:52.377986 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.377904    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:53.377892062 +0000 UTC m=+3.133305118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:52.377986 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.377928    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:53.37791065 +0000 UTC m=+3.133323722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:52.478933 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:52.478842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:52.479097 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.478994    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:52.479097 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.479014    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:52.479097 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.479026    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:52.479097 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:52.479083    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:53.479064809 +0000 UTC m=+3.234477869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:54.253636 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.253573    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:29:53 +0000 UTC" deadline="2028-02-03 14:43:32.910814292 +0000 UTC"
Jul 28 12:34:54.253636 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.253632    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13322h8m38.657186387s"
Jul 28 12:34:54.297371 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.297294    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:54.339996 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.339970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:54.340157 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:54.340117    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:34:54.340214 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.340169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:54.340335 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:54.340291    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:34:54.369733 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.369632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerStarted","Data":"0c0d654192825f6bd4304f8d0d7c412307f67624934ad6a9f24610934376acf4"}
Jul 28 12:34:54.374444 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.374412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mw6vl" event={"ID":"bb855a11-5572-4be0-a3ba-a0830feedecf","Type":"ContainerStarted","Data":"e46ba3517ac0f4d5ad30a35bab043afb0c21a32936a96cf7f8e25da9bcd9cd26"}
Jul 28 12:34:54.380300 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.380246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cptn6" event={"ID":"d62d21db-38dd-451a-967d-91a67fa38c39","Type":"ContainerStarted","Data":"07481522c3e899cc4058ccf2baa23f03d91eba74dd3ba997cf61b52dfdb7fc03"}
Jul 28 12:34:54.398544 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.398508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"d5e85a1f31926dcdfb97a85251edfad21fbb7435175500d002824a397deadbf7"}
Jul 28 12:34:54.414875 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.414840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wrq8" event={"ID":"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049","Type":"ContainerStarted","Data":"4f4469df5c1230db2b0b9db2f93edc7d134fc4d636c57df98a6c7b911f35bc47"}
Jul 28 12:34:54.428944 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.428912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal" event={"ID":"e0ede8a7a9a486a43a28960227e9b07d","Type":"ContainerStarted","Data":"48870ea590b4bb357708eceafdef20d7a16621182006e4f4c98550bd48e354d9"}
Jul 28 12:34:54.436771 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.436739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv" event={"ID":"5b10be79-9adb-469e-8d70-776c649328f9","Type":"ContainerStarted","Data":"debf2fa5820088da50b1846d65ce0cafdf63af09833ca6952d24dfc030f7bd5c"}
Jul 28 12:34:54.444793 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.444762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57pb8" event={"ID":"343e89fd-3c76-4dcd-8154-d28f1ba2daec","Type":"ContainerStarted","Data":"ebf7e3cf4b0cf1f8383bb4608672281c5db5ca9df64d11b9e9e2a1993a11e2d3"}
Jul 28 12:34:54.449559 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.449533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vz9w" event={"ID":"71572878-eed8-4109-913e-eee642504dba","Type":"ContainerStarted","Data":"9361fe71e1418be6b38375ee05ce3c269b823e9da6dd43061e55175f509072e5"}
Jul 28 12:34:54.459279 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.459255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" event={"ID":"81197a6b43549ebbb8245f4ccb902654","Type":"ContainerStarted","Data":"3d648727fa321b9d14ea7093989b02dc4e509f3784a7201955e2f24fc41da097"}
Jul 28 12:34:54.467566 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.467541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5bdt" event={"ID":"f8db7ce2-f506-453a-8d17-36c233734369","Type":"ContainerStarted","Data":"b631087b2a3fe6f138d3eca3a1e9591401d5999800ed41d4d16a15abdc9f1171"}
Jul 28 12:34:54.923717 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.923682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:54.924001 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:54.923735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:54.924001 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:54.923919    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:54.924001 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:54.923982    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:56.923963427 +0000 UTC m=+5.142316120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:54.924396 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:54.924377    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:54.924494 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:54.924431    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:56.924414932 +0000 UTC m=+5.142767626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:55.024154 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:55.024119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:55.024341 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:55.024290    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:55.024341 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:55.024312    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:55.024341 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:55.024325    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:55.024507 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:55.024386    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:34:57.024366547 +0000 UTC m=+5.242719238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:55.254299 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:55.254172    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:29:53 +0000 UTC" deadline="2028-01-01 11:20:40.98246846 +0000 UTC"
Jul 28 12:34:55.254299 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:55.254216    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12526h45m45.728257018s"
Jul 28 12:34:55.339507 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:55.339473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:55.339691 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:55.339606    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:34:56.118911 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:56.118879    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:34:56.343065 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:56.342997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:56.343478 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:56.343156    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:34:56.343700 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:56.343670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:56.343811 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:56.343790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:34:56.946668 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:56.946635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:56.946868 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:56.946687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:56.946940 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:56.946875    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:56.946999 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:56.946943    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:00.94692177 +0000 UTC m=+9.165274484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:34:56.947371 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:56.947352    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:56.947454 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:56.947408    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:00.947392456 +0000 UTC m=+9.165745152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:34:57.048041 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:57.047423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:57.048041 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:57.047632    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:34:57.048041 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:57.047653    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:34:57.048041 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:57.047665    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:57.048041 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:57.047713    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:01.047697562 +0000 UTC m=+9.266050252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:34:57.339085 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:57.338996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:57.339252 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:57.339127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:34:58.339979 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:58.339942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:34:58.340441 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:58.340091    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:34:58.340519 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:58.340499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:34:58.340654 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:58.340602    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:34:59.339350 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:34:59.339316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:34:59.339551 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:34:59.339450    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:00.341528 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:00.341494    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:00.342017 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:00.341639    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:00.342081 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:00.342054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:00.342179 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:00.342146    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:00.982496 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:00.982463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:00.982496 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:00.982501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:00.982699 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:00.982655    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:00.982699 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:00.982683    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:00.982793 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:00.982748    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:08.982729151 +0000 UTC m=+17.201081866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:00.982793 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:00.982786    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:08.982765944 +0000 UTC m=+17.201118640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:01.083939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:01.083893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:01.084101 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:01.084055    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:01.084101 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:01.084074    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:01.084101 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:01.084086    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:01.084262 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:01.084160    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:09.084142999 +0000 UTC m=+17.302495705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:01.339383 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:01.339299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:01.339538 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:01.339438    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:02.340580 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:02.340551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:02.341039 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:02.340690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:02.341039 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:02.340771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:02.341039 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:02.340878    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:03.339169 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:03.339131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:03.339353 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:03.339250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:04.339021 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:04.338988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:04.339580 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:04.339555    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:04.340009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:04.339963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:04.340721 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:04.340211    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:05.339119 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:05.339088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:05.339483 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:05.339182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:06.339042 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:06.339007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:06.339212 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:06.339005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:06.339212 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:06.339144    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:06.339581 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:06.339221    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:07.339716 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:07.339675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:07.340103 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:07.339790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:08.339716 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:08.339675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:08.339915 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:08.339800    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:08.339915 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:08.339864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:08.340336 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:08.339981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:09.039445 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:09.039405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:09.039684 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:09.039506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:09.039684 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.039582    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:09.039684 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.039599    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:09.039684 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.039664    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:25.039634294 +0000 UTC m=+33.257986988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:09.039684 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.039687    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:25.039675211 +0000 UTC m=+33.258027905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:09.139880 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:09.139846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:09.140025 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.139989    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:09.140025 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.140006    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:09.140025 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.140014    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:09.140121 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.140068    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:25.140051864 +0000 UTC m=+33.358404554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:09.339140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:09.339075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:09.339260 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:09.339169    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:10.338921 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:10.338887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:10.339359 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:10.338888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:10.339359 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:10.339002    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:10.339359 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:10.339103    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:11.339350 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:11.339318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:11.339765 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:11.339425    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:12.341104 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:12.340665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:12.341104 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:12.340728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:12.341104 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:12.341057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:12.341668 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:12.341121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:13.339442 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:13.339410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:13.339611 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:13.339518    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:14.339423 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.339393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:14.339811 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.339393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:14.339811 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:14.339500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:14.339811 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:14.339594    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:14.517532 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.517466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal" event={"ID":"e0ede8a7a9a486a43a28960227e9b07d","Type":"ContainerStarted","Data":"3720999f4ea5ecc38958aca4582db43315a47888cd10a386922bbd7b0dbbe95d"}
Jul 28 12:35:14.520613 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.520582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5bdt" event={"ID":"f8db7ce2-f506-453a-8d17-36c233734369","Type":"ContainerStarted","Data":"a47150eb19db18fd266b635a7fbf889e0ca1df87117bdeed0b94ef6d114c1c20"}
Jul 28 12:35:14.522722 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.522700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mw6vl" event={"ID":"bb855a11-5572-4be0-a3ba-a0830feedecf","Type":"ContainerStarted","Data":"fbf59ddab33d2236e4212004224e548523ec581e38a236994cbb61fefd0ef1d9"}
Jul 28 12:35:14.526167 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.526150    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:35:14.526596 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.526576    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7b35311-e67c-403b-afd8-7755a95f64a5" containerID="d19d2c5272036f22b327341a09bfeba136d2f16f608148b7e7cdb1b9595f7f36" exitCode=1
Jul 28 12:35:14.526689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.526610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerDied","Data":"d19d2c5272036f22b327341a09bfeba136d2f16f608148b7e7cdb1b9595f7f36"}
Jul 28 12:35:14.526689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.526681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"1cb6bf4153c7cc091b15aa9574011ffd634ee5fd831f7739d9e9c076a4a46415"}
Jul 28 12:35:14.532963 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.532919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-92.ec2.internal" podStartSLOduration=21.532904932 podStartE2EDuration="21.532904932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:14.532639814 +0000 UTC m=+22.750992529" watchObservedRunningTime="2026-07-28 12:35:14.532904932 +0000 UTC m=+22.751257645"
Jul 28 12:35:14.572789 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:14.570993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5bdt" podStartSLOduration=2.531473505 podStartE2EDuration="22.570975421s" totalImagesPullingTime="20.039501916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:14.551974913 +0000 UTC m=+22.770327626" watchObservedRunningTime="2026-07-28 12:35:14.570975421 +0000 UTC m=+22.789328127"
Jul 28 12:35:15.339530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.339341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:15.340302 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:15.339612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:15.530520 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.530481    2563 generic.go:356] "Generic (PLEG): container finished" podID="81197a6b43549ebbb8245f4ccb902654" containerID="5ac71f017a696274c0cf2ec4a931f33e3bf5dd9f79517b96cb0372c0d9ff59b4" exitCode=0
Jul 28 12:35:15.530741 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.530569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" event={"ID":"81197a6b43549ebbb8245f4ccb902654","Type":"ContainerDied","Data":"5ac71f017a696274c0cf2ec4a931f33e3bf5dd9f79517b96cb0372c0d9ff59b4"}
Jul 28 12:35:15.532280 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.532254    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6" containerID="14fbc3d8addba5595fd63178d9573537c2eaf42f1f2c6de755d0522466fe211d" exitCode=0
Jul 28 12:35:15.532417 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.532341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerDied","Data":"14fbc3d8addba5595fd63178d9573537c2eaf42f1f2c6de755d0522466fe211d"}
Jul 28 12:35:15.534549 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.534523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cptn6" event={"ID":"d62d21db-38dd-451a-967d-91a67fa38c39","Type":"ContainerStarted","Data":"1387c39f48895c5447be9360df0827113a2193d81abaa9aa4e310766f6689801"}
Jul 28 12:35:15.537799 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.537781    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:35:15.538214 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.538194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"53cca87073c8c4e1405870c51964ad0ac152e950c3c3433d68967c95f5b8cedf"}
Jul 28 12:35:15.538315 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.538268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"ffbda5018134c8b912aa7935ea6a663002159c3671c0cf89251e8964cbf21eb1"}
Jul 28 12:35:15.538359 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.538322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"718bbd1893efc6bc3df7b05909b03dc03e0fefc1bb1671a3cab5e90c14cef24b"}
Jul 28 12:35:15.538402 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.538369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"8a77eb48e812dec704fcb08245cb9ee7903149490d4fe386b2dc73c5bff525c9"}
Jul 28 12:35:15.539861 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.539837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wrq8" event={"ID":"5cd0dcaa-46fa-4da7-a97e-c8737a1bf049","Type":"ContainerStarted","Data":"19802347b8a52bd0327bd05c5593f5e48aa75daf4d602146ab3e90d90b7fdcab"}
Jul 28 12:35:15.541311 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.541284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv" event={"ID":"5b10be79-9adb-469e-8d70-776c649328f9","Type":"ContainerStarted","Data":"20c96f15fb9b95bac90f131868f9bbb2525af99eda9a733674465663d7ed8a8c"}
Jul 28 12:35:15.542666 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.542641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57pb8" event={"ID":"343e89fd-3c76-4dcd-8154-d28f1ba2daec","Type":"ContainerStarted","Data":"7a2bd567f12802438f35de4f3521b27abd0f5eaa80b84c2e01a21d5e85391ff4"}
Jul 28 12:35:15.544131 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.544110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vz9w" event={"ID":"71572878-eed8-4109-913e-eee642504dba","Type":"ContainerStarted","Data":"d6c73d20362b8a8f907fbe612d7b44260b8dc38d18867cc4929cc0ed46d42f4f"}
Jul 28 12:35:15.546765 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.546730    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mw6vl" podStartSLOduration=3.543442432 podStartE2EDuration="23.546719118s" totalImagesPullingTime="20.003276686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:14.572882087 +0000 UTC m=+22.791234799" watchObservedRunningTime="2026-07-28 12:35:15.546719118 +0000 UTC m=+23.765071829"
Jul 28 12:35:15.567178 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.567137    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6wrq8" podStartSLOduration=3.563463618 podStartE2EDuration="23.567124582s" totalImagesPullingTime="20.003660964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.560013955 +0000 UTC m=+23.778366667" watchObservedRunningTime="2026-07-28 12:35:15.567124582 +0000 UTC m=+23.785477293"
Jul 28 12:35:15.574850 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.574781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-57pb8" podStartSLOduration=3.482423163 podStartE2EDuration="23.574765843s" totalImagesPullingTime="20.09234268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.574126232 +0000 UTC m=+23.792478946" watchObservedRunningTime="2026-07-28 12:35:15.574765843 +0000 UTC m=+23.793118555"
Jul 28 12:35:15.609673 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.609627    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4vz9w" podStartSLOduration=3.499854834 podStartE2EDuration="23.609611953s" totalImagesPullingTime="20.109757119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.609199823 +0000 UTC m=+23.827552535" watchObservedRunningTime="2026-07-28 12:35:15.609611953 +0000 UTC m=+23.827964665"
Jul 28 12:35:15.986677 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:15.986650    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:35:16.268562 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.268465    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:35:15.986671445Z","UUID":"cdb40bbb-fe8a-4186-8209-e741800f6303","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:35:16.270142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.270119    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:35:16.270266 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.270166    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:35:16.299070 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.299031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:35:16.299691 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.299667    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:35:16.320285 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.320216    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cptn6" podStartSLOduration=4.272622495 podStartE2EDuration="24.320198451s" totalImagesPullingTime="20.047575956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:15.623805182 +0000 UTC m=+23.842157893" watchObservedRunningTime="2026-07-28 12:35:16.320198451 +0000 UTC m=+24.538551164"
Jul 28 12:35:16.338880 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.338860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:16.339018 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:16.338962    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:16.339074 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.339027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:16.339160 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:16.339140    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:16.548708 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.548622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv" event={"ID":"5b10be79-9adb-469e-8d70-776c649328f9","Type":"ContainerStarted","Data":"a206941d8550d3cc8ffef691efb337d9b2552d1a76ae923ae0f104ce492d99c4"}
Jul 28 12:35:16.550642 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.550603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" event={"ID":"81197a6b43549ebbb8245f4ccb902654","Type":"ContainerStarted","Data":"42789bc2caca24e70a4946ec18822421e23a760471465c8623ef93fe49e51616"}
Jul 28 12:35:16.551269 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.551233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:35:16.551518 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.551502    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cptn6"
Jul 28 12:35:16.569660 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:16.569614    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-92.ec2.internal" podStartSLOduration=23.569597191 podStartE2EDuration="23.569597191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:16.569453554 +0000 UTC m=+24.787806267" watchObservedRunningTime="2026-07-28 12:35:16.569597191 +0000 UTC m=+24.787949902"
Jul 28 12:35:17.339610 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:17.339582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:17.339779 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:17.339706    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:17.556032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:17.556001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:35:17.556466 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:17.556436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"059f300d6fea1677f2d1f331388eb6a806e52bceca00205127213dddbbd2d8d8"}
Jul 28 12:35:18.342744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:18.342669    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:18.342924 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:18.342795    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:18.342924 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:18.342856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:18.343037 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:18.342958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:19.339483 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:19.339401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:19.339881 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:19.339534    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:20.339054 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.338864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:20.339330 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.338872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:20.339330 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:20.339166    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:20.339330 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:20.339215    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:20.564721 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.564692    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6" containerID="7eb61f5d0b5afd3a5cb7b978c62e4aabd2ddc59bd722aad685efaccd6a3ad6f6" exitCode=0
Jul 28 12:35:20.565291 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.564765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerDied","Data":"7eb61f5d0b5afd3a5cb7b978c62e4aabd2ddc59bd722aad685efaccd6a3ad6f6"}
Jul 28 12:35:20.568376 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.568360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:35:20.568783 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.568750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"dadcdf69960d21fcaef6f71256f9a35c4416d0a0f5bc39ba42e6cfd471fce8c8"}
Jul 28 12:35:20.568988 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.568973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:35:20.569094 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.569072    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:35:20.569210 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.569195    2563 scope.go:117] "RemoveContainer" containerID="d19d2c5272036f22b327341a09bfeba136d2f16f608148b7e7cdb1b9595f7f36"
Jul 28 12:35:20.571489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.571470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv" event={"ID":"5b10be79-9adb-469e-8d70-776c649328f9","Type":"ContainerStarted","Data":"af6f7fd2673eed7437b9b27c9316aa739960bd636e204d0f20c82db2314b9969"}
Jul 28 12:35:20.588284 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.588263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:35:20.604395 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:20.604359    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2bfv" podStartSLOduration=3.564955362 podStartE2EDuration="28.604345426s" totalImagesPullingTime="25.039390064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:20.604178369 +0000 UTC m=+28.822531081" watchObservedRunningTime="2026-07-28 12:35:20.604345426 +0000 UTC m=+28.822698138"
Jul 28 12:35:21.339332 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:21.339071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:21.339682 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:21.339382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:21.577422 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:21.577396    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:35:21.577886 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:21.577856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" event={"ID":"b7b35311-e67c-403b-afd8-7755a95f64a5","Type":"ContainerStarted","Data":"f248999d414116096f77d1f1a0dcf98c37881cb021c1c553c106fe3bb4fbe623"}
Jul 28 12:35:21.578229 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:21.578202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:35:21.593810 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:21.593755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:35:21.611204 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:21.611160    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz" podStartSLOduration=8.943646583 podStartE2EDuration="29.611145504s" totalImagesPullingTime="20.667498921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:21.609688983 +0000 UTC m=+29.828041696" watchObservedRunningTime="2026-07-28 12:35:21.611145504 +0000 UTC m=+29.829498217"
Jul 28 12:35:22.099254 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.099178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dh4h9"]
Jul 28 12:35:22.099488 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.099288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:22.099488 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:22.099372    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:22.102062 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.102037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tsz9"]
Jul 28 12:35:22.102189 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.102135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:22.102247 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:22.102214    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:22.102622 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.102603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqk86"]
Jul 28 12:35:22.102724 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.102710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:22.102851 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:22.102814    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:22.581772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.581736    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6" containerID="8f8109c3ede9fb341062cfef6ec107839ce72d5ab51bdaa2e1611dbb151d4c68" exitCode=0
Jul 28 12:35:22.582181 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:22.581837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerDied","Data":"8f8109c3ede9fb341062cfef6ec107839ce72d5ab51bdaa2e1611dbb151d4c68"}
Jul 28 12:35:23.339697 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:23.339661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:23.339890 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:23.339779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:23.339890 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:23.339842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:23.340012 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:23.339931    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:24.339709 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:24.339679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:24.340130 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:24.339779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:24.587855 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:24.587804    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6" containerID="0e42149ca255b76cf208e0032b6985c5fcf1799c5006adb42df785c9cda3875b" exitCode=0
Jul 28 12:35:24.588020 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:24.587887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerDied","Data":"0e42149ca255b76cf208e0032b6985c5fcf1799c5006adb42df785c9cda3875b"}
Jul 28 12:35:25.059097 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:25.059068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:25.059097 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:25.059105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:25.059301 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.059212    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:25.059301 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.059272    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:57.059256252 +0000 UTC m=+65.277608942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:25.059301 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.059217    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:25.059301 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.059302    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret podName:30fd4493-5528-42f8-b77e-a73c763a2116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:57.059295947 +0000 UTC m=+65.277648637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret") pod "global-pull-secret-syncer-dh4h9" (UID: "30fd4493-5528-42f8-b77e-a73c763a2116") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:25.160078 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:25.160049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:25.160206 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.160172    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:25.160206 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.160185    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:25.160206 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.160193    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wv2vw for pod openshift-network-diagnostics/network-check-target-5tsz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:25.160314 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.160239    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw podName:76dd0f1f-9e03-4e9c-8e32-456992cda116 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:57.160226341 +0000 UTC m=+65.378579030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wv2vw" (UniqueName: "kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw") pod "network-check-target-5tsz9" (UID: "76dd0f1f-9e03-4e9c-8e32-456992cda116") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:25.339096 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:25.339017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:25.339240 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.339125    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:25.339240 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:25.339176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:25.339326 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:25.339264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:26.339284 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:26.339253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:26.339690 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:26.339366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4h9" podUID="30fd4493-5528-42f8-b77e-a73c763a2116"
Jul 28 12:35:27.339732 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:27.339702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:27.340187 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:27.339698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:27.340187 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:27.339859    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqk86" podUID="7ae2f41f-73c2-451c-b9fe-c6a94c59a238"
Jul 28 12:35:27.340187 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:27.339915    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tsz9" podUID="76dd0f1f-9e03-4e9c-8e32-456992cda116"
Jul 28 12:35:28.083896 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.083683    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-92.ec2.internal" event="NodeReady"
Jul 28 12:35:28.084071 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.084008    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:35:28.125043 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.124994    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7698788d7d-6chn2"]
Jul 28 12:35:28.127532 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.127513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.129864 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.129842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:35:28.130339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.130310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:35:28.130454 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.130349    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w2rc8\""
Jul 28 12:35:28.130533 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.130510    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:35:28.137550 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.137531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:35:28.143942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.143921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7698788d7d-6chn2"]
Jul 28 12:35:28.146897 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.146875    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-clr4g"]
Jul 28 12:35:28.148944 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.148924    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jnrmn"]
Jul 28 12:35:28.149062 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.149047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.150861 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.150842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.152557 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.152539    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:35:28.156086 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.156067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:35:28.156199 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.156168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:35:28.156314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.156275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:35:28.157022 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.157006    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l7wd\""
Jul 28 12:35:28.157560 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.157541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:35:28.157783 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.157769    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k2njb\""
Jul 28 12:35:28.175793 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.175761    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jnrmn"]
Jul 28 12:35:28.184364 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.184344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clr4g"]
Jul 28 12:35:28.279036 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-image-registry-private-configuration\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/217ed5d2-02b2-4ca3-95a0-50d207754e72-ca-trust-extracted\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxbm\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-kube-api-access-7gxbm\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18018de2-4df9-4e7b-bbe9-6aab7abf990a-cert\") pod \"ingress-canary-jnrmn\" (UID: \"18018de2-4df9-4e7b-bbe9-6aab7abf990a\") " pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-tmp-dir\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-certificates\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-trusted-ca\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-bound-sa-token\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-config-volume\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.279489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvtj\" (UniqueName: \"kubernetes.io/projected/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-kube-api-access-bdvtj\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.279489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrg2c\" (UniqueName: \"kubernetes.io/projected/18018de2-4df9-4e7b-bbe9-6aab7abf990a-kube-api-access-xrg2c\") pod \"ingress-canary-jnrmn\" (UID: \"18018de2-4df9-4e7b-bbe9-6aab7abf990a\") " pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.279489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-installation-pull-secrets\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.279489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.279440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.339357 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.339288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:28.341971 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.341937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:35:28.370551 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.370508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cptn6_d62d21db-38dd-451a-967d-91a67fa38c39/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:28.380360 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-installation-pull-secrets\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.380455 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.380518 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-image-registry-private-configuration\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.380518 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/217ed5d2-02b2-4ca3-95a0-50d207754e72-ca-trust-extracted\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.380614 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxbm\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-kube-api-access-7gxbm\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.380614 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18018de2-4df9-4e7b-bbe9-6aab7abf990a-cert\") pod \"ingress-canary-jnrmn\" (UID: \"18018de2-4df9-4e7b-bbe9-6aab7abf990a\") " pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.380614 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-tmp-dir\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.380848 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-certificates\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.380913 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-trusted-ca\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.380913 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-bound-sa-token\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.381009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-config-volume\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.381009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.381009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.380992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvtj\" (UniqueName: \"kubernetes.io/projected/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-kube-api-access-bdvtj\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.381156 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.381020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrg2c\" (UniqueName: \"kubernetes.io/projected/18018de2-4df9-4e7b-bbe9-6aab7abf990a-kube-api-access-xrg2c\") pod \"ingress-canary-jnrmn\" (UID: \"18018de2-4df9-4e7b-bbe9-6aab7abf990a\") " pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.381156 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.381038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-tmp-dir\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.381156 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.381105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/217ed5d2-02b2-4ca3-95a0-50d207754e72-ca-trust-extracted\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.381156 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.381115    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:28.381345 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.381177    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:35:28.88115985 +0000 UTC m=+37.099512564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:28.381411 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.381373    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:28.381411 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.381387    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:35:28.381514 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.381430    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:28.881417176 +0000 UTC m=+37.099769870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:35:28.381746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.381722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-config-volume\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.382009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.381988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-certificates\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.382411 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.382386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-trusted-ca\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.385638 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.385617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-installation-pull-secrets\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.385737 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.385651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18018de2-4df9-4e7b-bbe9-6aab7abf990a-cert\") pod \"ingress-canary-jnrmn\" (UID: \"18018de2-4df9-4e7b-bbe9-6aab7abf990a\") " pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.386189 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.386169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-image-registry-private-configuration\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.390689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.390668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxbm\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-kube-api-access-7gxbm\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.390798 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.390777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-bound-sa-token\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.391611 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.391591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvtj\" (UniqueName: \"kubernetes.io/projected/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-kube-api-access-bdvtj\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.393703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.393683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrg2c\" (UniqueName: \"kubernetes.io/projected/18018de2-4df9-4e7b-bbe9-6aab7abf990a-kube-api-access-xrg2c\") pod \"ingress-canary-jnrmn\" (UID: \"18018de2-4df9-4e7b-bbe9-6aab7abf990a\") " pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.464804 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.464770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jnrmn"
Jul 28 12:35:28.466061 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.466013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cptn6_d62d21db-38dd-451a-967d-91a67fa38c39/konnectivity-agent/0.log" limit=1024
Jul 28 12:35:28.884948 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.884919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:28.885131 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:28.885013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:28.885131 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.885100    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:28.885131 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.885103    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:28.885131 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.885111    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:35:28.885312 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.885181    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:35:29.885159699 +0000 UTC m=+38.103512389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:28.885312 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:28.885201    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:29.885191848 +0000 UTC m=+38.103544542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:35:29.340469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.339716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:29.340469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.339765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:29.342388 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.342367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:35:29.342736 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.342395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kxg2r\""
Jul 28 12:35:29.343170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.343146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:35:29.343310 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.343230    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:35:29.343395 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.343375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z4tgw\""
Jul 28 12:35:29.669265 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.669041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jnrmn"]
Jul 28 12:35:29.673612 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:35:29.673438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18018de2_4df9_4e7b_bbe9_6aab7abf990a.slice/crio-3f8525e8e94331da7b83a7715c558b50a0a6ce429bd9e23d09e52ba6de18790c WatchSource:0}: Error finding container 3f8525e8e94331da7b83a7715c558b50a0a6ce429bd9e23d09e52ba6de18790c: Status 404 returned error can't find the container with id 3f8525e8e94331da7b83a7715c558b50a0a6ce429bd9e23d09e52ba6de18790c
Jul 28 12:35:29.891990 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.891867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:29.891990 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:29.891932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:29.892195 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:29.892044    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:29.892195 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:29.892064    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:35:29.892195 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:29.892090    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:29.892195 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:29.892120    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:31.892104077 +0000 UTC m=+40.110456771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:35:29.892195 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:29.892141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:35:31.892127064 +0000 UTC m=+40.110479755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:30.606325 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:30.606287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jnrmn" event={"ID":"18018de2-4df9-4e7b-bbe9-6aab7abf990a","Type":"ContainerStarted","Data":"3f8525e8e94331da7b83a7715c558b50a0a6ce429bd9e23d09e52ba6de18790c"}
Jul 28 12:35:30.609517 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:30.609486    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6" containerID="cc91873295fd877c2c8a7ecd30ec786cb948d5b7c4609bfe130f5c98bb1ced9d" exitCode=0
Jul 28 12:35:30.609635 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:30.609527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerDied","Data":"cc91873295fd877c2c8a7ecd30ec786cb948d5b7c4609bfe130f5c98bb1ced9d"}
Jul 28 12:35:31.616339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:31.616290    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6" containerID="7867fdb2108b3516070cb5a595a3a5333f96dd688b113ec575be2e335e48e9ca" exitCode=0
Jul 28 12:35:31.616339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:31.616328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerDied","Data":"7867fdb2108b3516070cb5a595a3a5333f96dd688b113ec575be2e335e48e9ca"}
Jul 28 12:35:31.908580 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:31.908508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:31.908580 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:31.908562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:31.908733 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:31.908655    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:31.908733 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:31.908679    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:35:31.908733 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:31.908730    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:35.908713785 +0000 UTC m=+44.127066475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:35:31.908868 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:31.908658    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:31.908868 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:31.908783    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:35:35.908772784 +0000 UTC m=+44.127125474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:32.622666 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:32.622634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" event={"ID":"e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6","Type":"ContainerStarted","Data":"ac391a9969540e732c09772076572312d3672db37ed568bf721a4d76aece5d9d"}
Jul 28 12:35:32.624079 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:32.624006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jnrmn" event={"ID":"18018de2-4df9-4e7b-bbe9-6aab7abf990a","Type":"ContainerStarted","Data":"3c81c849f97678e0e8ecafece47c9a502129fd8691ab716c694cd5b08f2af830"}
Jul 28 12:35:32.648794 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:32.648756    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9gc9r" podStartSLOduration=9.159891231 podStartE2EDuration="40.648741949s" totalImagesPullingTime="31.488850718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:32.647065661 +0000 UTC m=+40.865418372" watchObservedRunningTime="2026-07-28 12:35:32.648741949 +0000 UTC m=+40.867094661"
Jul 28 12:35:32.665339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:32.665296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jnrmn" podStartSLOduration=1.966640833 podStartE2EDuration="4.665282225s" totalImagesPullingTime="2.698641392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:32.664712099 +0000 UTC m=+40.883064803" watchObservedRunningTime="2026-07-28 12:35:32.665282225 +0000 UTC m=+40.883634933"
Jul 28 12:35:35.943628 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:35.943588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:35.944050 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:35.943659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:35.944050 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:35.943745    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:35.944050 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:35.943767    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:35:35.944050 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:35.943796    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:35.944050 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:35.943845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:43.943808606 +0000 UTC m=+52.162161299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:35:35.944050 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:35.943864    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:35:43.943852142 +0000 UTC m=+52.162204835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:43.999250 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:43.999211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:35:43.999662 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:43.999264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:35:43.999662 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:43.999375    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:43.999662 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:43.999374    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:35:43.999662 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:43.999400    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:35:43.999662 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:43.999430    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:35:59.999413617 +0000 UTC m=+68.217766307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:43.999662 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:43.999493    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:59.999473074 +0000 UTC m=+68.217825767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:35:47.382551 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.382515    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"]
Jul 28 12:35:47.403322 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.403304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.408351 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.408323    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:35:47.408474 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.408424    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:35:47.408665 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.408646    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 12:35:47.408851 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.408681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br6h9\""
Jul 28 12:35:47.408932 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.408700    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 12:35:47.409369 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.409349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"]
Jul 28 12:35:47.431431 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.431408    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"]
Jul 28 12:35:47.451507 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.451486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"]
Jul 28 12:35:47.451589 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.451561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.456147 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.456120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 12:35:47.456251 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.456233    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 12:35:47.456317 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.456287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:47.456317 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.456248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ll2pv\""
Jul 28 12:35:47.456610 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.456595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:47.526782 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.526760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.526893 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.526788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rjh\" (UniqueName: \"kubernetes.io/projected/d14de7a2-2981-41a0-b3c2-980fd5ba681b-kube-api-access-p4rjh\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.526933 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.526921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d14de7a2-2981-41a0-b3c2-980fd5ba681b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.628257 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d14de7a2-2981-41a0-b3c2-980fd5ba681b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.628373 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshg5\" (UniqueName: \"kubernetes.io/projected/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-kube-api-access-cshg5\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.628373 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.628373 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.628527 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-config\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.628527 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:47.628399    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:47.628527 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:47.628466    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls podName:d14de7a2-2981-41a0-b3c2-980fd5ba681b nodeName:}" failed. No retries permitted until 2026-07-28 12:35:48.128444514 +0000 UTC m=+56.346797208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-q9znf" (UID: "d14de7a2-2981-41a0-b3c2-980fd5ba681b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:47.628527 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rjh\" (UniqueName: \"kubernetes.io/projected/d14de7a2-2981-41a0-b3c2-980fd5ba681b-kube-api-access-p4rjh\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.628931 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.628914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d14de7a2-2981-41a0-b3c2-980fd5ba681b-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.641326 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.641273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rjh\" (UniqueName: \"kubernetes.io/projected/d14de7a2-2981-41a0-b3c2-980fd5ba681b-kube-api-access-p4rjh\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:47.729131 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.729096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.729131 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.729130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-config\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.729321 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.729206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshg5\" (UniqueName: \"kubernetes.io/projected/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-kube-api-access-cshg5\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.729844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.729747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-config\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.731379 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.731355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.743916 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.743891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshg5\" (UniqueName: \"kubernetes.io/projected/ed5a6cde-92cb-49ff-bba3-41f0a7e897fd-kube-api-access-cshg5\") pod \"service-ca-operator-5d6c75f84f-lld9z\" (UID: \"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.759776 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.759752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"
Jul 28 12:35:47.885247 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:47.885217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z"]
Jul 28 12:35:47.887971 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:35:47.887944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5a6cde_92cb_49ff_bba3_41f0a7e897fd.slice/crio-ba3a69fae6910362463e14f6c4d1e6d93bbfce1d468ae6fb05d880cdf572955f WatchSource:0}: Error finding container ba3a69fae6910362463e14f6c4d1e6d93bbfce1d468ae6fb05d880cdf572955f: Status 404 returned error can't find the container with id ba3a69fae6910362463e14f6c4d1e6d93bbfce1d468ae6fb05d880cdf572955f
Jul 28 12:35:48.132410 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:48.132373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:48.132585 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:48.132529    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:48.132634 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:48.132594    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls podName:d14de7a2-2981-41a0-b3c2-980fd5ba681b nodeName:}" failed. No retries permitted until 2026-07-28 12:35:49.132579306 +0000 UTC m=+57.350932000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-q9znf" (UID: "d14de7a2-2981-41a0-b3c2-980fd5ba681b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:48.663207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:48.663159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z" event={"ID":"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd","Type":"ContainerStarted","Data":"ba3a69fae6910362463e14f6c4d1e6d93bbfce1d468ae6fb05d880cdf572955f"}
Jul 28 12:35:49.139861 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:49.139800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:49.140032 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:49.139974    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:49.140092 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:49.140048    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls podName:d14de7a2-2981-41a0-b3c2-980fd5ba681b nodeName:}" failed. No retries permitted until 2026-07-28 12:35:51.140032169 +0000 UTC m=+59.358384859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-q9znf" (UID: "d14de7a2-2981-41a0-b3c2-980fd5ba681b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:49.666811 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:49.666726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z" event={"ID":"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd","Type":"ContainerStarted","Data":"6ebc513adb6885b5b4e3f36d28686aaa6c254b3389892f240995e1c2050c6698"}
Jul 28 12:35:49.683794 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:49.683744    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z" podStartSLOduration=1.228553336 podStartE2EDuration="2.683730605s" totalImagesPullingTime="1.455177269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:49.682916573 +0000 UTC m=+57.901269297" watchObservedRunningTime="2026-07-28 12:35:49.683730605 +0000 UTC m=+57.902083316"
Jul 28 12:35:51.154631 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:51.154585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:51.155036 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:51.154720    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:51.155036 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:51.154790    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls podName:d14de7a2-2981-41a0-b3c2-980fd5ba681b nodeName:}" failed. No retries permitted until 2026-07-28 12:35:55.154773513 +0000 UTC m=+63.373126204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-q9znf" (UID: "d14de7a2-2981-41a0-b3c2-980fd5ba681b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:52.562885 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.562853    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bb42n"]
Jul 28 12:35:52.565260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.565239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.568777 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.568756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:35:52.568930 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.568882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:35:52.568998 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.568936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:35:52.569057 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.568999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:35:52.569297 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.569280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nrjs7\""
Jul 28 12:35:52.585222 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.585199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bb42n"]
Jul 28 12:35:52.665347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.665317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-data-volume\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.665467 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.665361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-crio-socket\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.665467 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.665383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.665467 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.665414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b5h\" (UniqueName: \"kubernetes.io/projected/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-api-access-j6b5h\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.665581 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.665468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.765750 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.765719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.765948 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.765770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b5h\" (UniqueName: \"kubernetes.io/projected/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-api-access-j6b5h\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.765948 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.765883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.765948 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:52.765901    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:52.765948 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.765924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-data-volume\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.766161 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:52.765969    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:35:53.265949081 +0000 UTC m=+61.484301791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:52.766161 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.765999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-crio-socket\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.766161 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:52.766000    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:52.766161 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:52.766081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:35:53.266069254 +0000 UTC m=+61.484421947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:52.766314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.766206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-crio-socket\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.766314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.766273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-data-volume\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:52.777113 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:52.777086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b5h\" (UniqueName: \"kubernetes.io/projected/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-api-access-j6b5h\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:53.269713 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:53.269679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:53.269894 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:53.269779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:53.269894 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:53.269849    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:53.269894 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:53.269867    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:53.270000 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:53.269918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:35:54.269899907 +0000 UTC m=+62.488252602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:53.270000 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:53.269940    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:35:54.269927381 +0000 UTC m=+62.488280071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:53.602382 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:53.602311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7tvz"
Jul 28 12:35:54.277673 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:54.277634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:54.277841 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:54.277730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:54.277841 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:54.277777    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:54.277841 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:54.277796    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:54.277957 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:54.277861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:35:56.27784316 +0000 UTC m=+64.496195850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:54.277957 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:54.277878    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:35:56.277870848 +0000 UTC m=+64.496223538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:54.349712 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:54.349690    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vz9w_71572878-eed8-4109-913e-eee642504dba/dns-node-resolver/0.log"
Jul 28 12:35:55.184763 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:55.184731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:35:55.185127 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:55.184889    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:55.185127 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:55.184954    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls podName:d14de7a2-2981-41a0-b3c2-980fd5ba681b nodeName:}" failed. No retries permitted until 2026-07-28 12:36:03.184938305 +0000 UTC m=+71.403290996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-q9znf" (UID: "d14de7a2-2981-41a0-b3c2-980fd5ba681b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:35:55.350028 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:55.349990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wrq8_5cd0dcaa-46fa-4da7-a97e-c8737a1bf049/node-ca/0.log"
Jul 28 12:35:56.150565 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:56.150534    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jnrmn_18018de2-4df9-4e7b-bbe9-6aab7abf990a/serve-healthcheck-canary/0.log"
Jul 28 12:35:56.293167 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:56.293121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:56.293585 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:56.293198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:35:56.293585 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:56.293242    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:56.293585 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:56.293303    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:36:00.293289751 +0000 UTC m=+68.511642441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:56.293585 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:56.293336    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:56.293585 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:56.293396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:36:00.293383768 +0000 UTC m=+68.511736461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:57.098790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.098748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:35:57.098790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.098792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:57.101576 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.101555    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:35:57.102279 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.102266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:35:57.109024 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:57.109007    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 12:35:57.109090 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:35:57.109069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs podName:7ae2f41f-73c2-451c-b9fe-c6a94c59a238 nodeName:}" failed. No retries permitted until 2026-07-28 12:37:01.109051051 +0000 UTC m=+129.327403741 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs") pod "network-metrics-daemon-dqk86" (UID: "7ae2f41f-73c2-451c-b9fe-c6a94c59a238") : secret "metrics-daemon-secret" not found
Jul 28 12:35:57.111289 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.111258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30fd4493-5528-42f8-b77e-a73c763a2116-original-pull-secret\") pod \"global-pull-secret-syncer-dh4h9\" (UID: \"30fd4493-5528-42f8-b77e-a73c763a2116\") " pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:57.149546 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.149523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4h9"
Jul 28 12:35:57.199900 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.199846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:57.202931 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.202911    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:35:57.213299 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.213239    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:35:57.223112 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.223086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2vw\" (UniqueName: \"kubernetes.io/projected/76dd0f1f-9e03-4e9c-8e32-456992cda116-kube-api-access-wv2vw\") pod \"network-check-target-5tsz9\" (UID: \"76dd0f1f-9e03-4e9c-8e32-456992cda116\") " pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:57.261856 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.261806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z4tgw\""
Jul 28 12:35:57.268409 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.268388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:35:57.283857 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.283810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dh4h9"]
Jul 28 12:35:57.287269 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:35:57.287249    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fd4493_5528_42f8_b77e_a73c763a2116.slice/crio-9b1d2d902c356773ccedaa70f736c067f1d2032ea642fce58348b3d9a3401752 WatchSource:0}: Error finding container 9b1d2d902c356773ccedaa70f736c067f1d2032ea642fce58348b3d9a3401752: Status 404 returned error can't find the container with id 9b1d2d902c356773ccedaa70f736c067f1d2032ea642fce58348b3d9a3401752
Jul 28 12:35:57.397961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.397889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tsz9"]
Jul 28 12:35:57.400713 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:35:57.400667    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dd0f1f_9e03_4e9c_8e32_456992cda116.slice/crio-c1e42dc7314e852c41512995f26deb9bc0f18dff0d068e0ec28e934b5d3e90b6 WatchSource:0}: Error finding container c1e42dc7314e852c41512995f26deb9bc0f18dff0d068e0ec28e934b5d3e90b6: Status 404 returned error can't find the container with id c1e42dc7314e852c41512995f26deb9bc0f18dff0d068e0ec28e934b5d3e90b6
Jul 28 12:35:57.685445 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.685361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dh4h9" event={"ID":"30fd4493-5528-42f8-b77e-a73c763a2116","Type":"ContainerStarted","Data":"9b1d2d902c356773ccedaa70f736c067f1d2032ea642fce58348b3d9a3401752"}
Jul 28 12:35:57.686483 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:35:57.686461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tsz9" event={"ID":"76dd0f1f-9e03-4e9c-8e32-456992cda116","Type":"ContainerStarted","Data":"c1e42dc7314e852c41512995f26deb9bc0f18dff0d068e0ec28e934b5d3e90b6"}
Jul 28 12:36:00.021412 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:00.021369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:36:00.021876 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:00.021445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:36:00.021876 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.021492    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 12:36:00.021876 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.021515    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7698788d7d-6chn2: secret "image-registry-tls" not found
Jul 28 12:36:00.021876 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.021583    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls podName:217ed5d2-02b2-4ca3-95a0-50d207754e72 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:32.021566652 +0000 UTC m=+100.239919342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls") pod "image-registry-7698788d7d-6chn2" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72") : secret "image-registry-tls" not found
Jul 28 12:36:00.021876 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.021597    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:36:00.021876 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.021645    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls podName:9f34d2d2-3bbb-4f64-95e4-e2087909f7cd nodeName:}" failed. No retries permitted until 2026-07-28 12:36:32.021627946 +0000 UTC m=+100.239980637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls") pod "dns-default-clr4g" (UID: "9f34d2d2-3bbb-4f64-95e4-e2087909f7cd") : secret "dns-default-metrics-tls" not found
Jul 28 12:36:00.324266 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:00.324171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:00.324435 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:00.324272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:00.324435 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.324312    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:36:00.324435 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.324373    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:00.324435 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.324410    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.324389291 +0000 UTC m=+76.542742004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : configmap "kube-rbac-proxy" not found
Jul 28 12:36:00.324435 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:00.324429    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.324416788 +0000 UTC m=+76.542769477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:02.702712 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:02.702667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dh4h9" event={"ID":"30fd4493-5528-42f8-b77e-a73c763a2116","Type":"ContainerStarted","Data":"3227d07655165097e4f175b841c4fbde2136ca8daae50f5a96ea77b1141868a3"}
Jul 28 12:36:02.704054 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:02.704032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tsz9" event={"ID":"76dd0f1f-9e03-4e9c-8e32-456992cda116","Type":"ContainerStarted","Data":"f22467ed40dde2ecc2353d28f844f9cda208bc939f93e432c08a984fafb990dc"}
Jul 28 12:36:02.704196 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:02.704159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:36:02.720415 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:02.720372    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dh4h9" podStartSLOduration=65.752338914 podStartE2EDuration="1m10.720359278s" totalImagesPullingTime="4.968020364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:02.719978908 +0000 UTC m=+70.938331623" watchObservedRunningTime="2026-07-28 12:36:02.720359278 +0000 UTC m=+70.938711967"
Jul 28 12:36:02.735779 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:02.735739    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5tsz9" podStartSLOduration=65.892262788 podStartE2EDuration="1m10.735726374s" totalImagesPullingTime="4.843463586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:02.735077509 +0000 UTC m=+70.953430215" watchObservedRunningTime="2026-07-28 12:36:02.735726374 +0000 UTC m=+70.954079064"
Jul 28 12:36:03.250722 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:03.250687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:36:03.250905 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:03.250857    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:36:03.250946 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:03.250924    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls podName:d14de7a2-2981-41a0-b3c2-980fd5ba681b nodeName:}" failed. No retries permitted until 2026-07-28 12:36:19.250909306 +0000 UTC m=+87.469261996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-q9znf" (UID: "d14de7a2-2981-41a0-b3c2-980fd5ba681b") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:36:08.388013 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:08.387984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:08.388460 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:08.388036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:08.388460 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:08.388149    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:08.388460 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:08.388198    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls podName:cb20ff7e-9864-4d09-b996-6a4f5e5c76ad nodeName:}" failed. No retries permitted until 2026-07-28 12:36:24.38818609 +0000 UTC m=+92.606538779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bb42n" (UID: "cb20ff7e-9864-4d09-b996-6a4f5e5c76ad") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:08.388574 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:08.388476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:13.230727 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.230689    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"]
Jul 28 12:36:13.232446 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.232422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.235598 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.235573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:36:13.236054 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.236037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dhrqc\""
Jul 28 12:36:13.236152 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.236078    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:36:13.236212 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.236181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:36:13.236212 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.236186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 12:36:13.253805 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.253784    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"]
Jul 28 12:36:13.323725 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.323698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a16d32ce-a015-47d1-bc1f-09e61cff8f77-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-858bd75c58-j5tgk\" (UID: \"a16d32ce-a015-47d1-bc1f-09e61cff8f77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.323897 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.323778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrcp\" (UniqueName: \"kubernetes.io/projected/a16d32ce-a015-47d1-bc1f-09e61cff8f77-kube-api-access-tbrcp\") pod \"managed-serviceaccount-addon-agent-858bd75c58-j5tgk\" (UID: \"a16d32ce-a015-47d1-bc1f-09e61cff8f77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.324743 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.324722    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"]
Jul 28 12:36:13.326469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.326452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"
Jul 28 12:36:13.329081 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.329065    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 12:36:13.329544 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.329530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gg7cx\""
Jul 28 12:36:13.330118 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.330105    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 12:36:13.344333 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.344309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"]
Jul 28 12:36:13.424216 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.424118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a16d32ce-a015-47d1-bc1f-09e61cff8f77-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-858bd75c58-j5tgk\" (UID: \"a16d32ce-a015-47d1-bc1f-09e61cff8f77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.424216 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.424179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrcp\" (UniqueName: \"kubernetes.io/projected/a16d32ce-a015-47d1-bc1f-09e61cff8f77-kube-api-access-tbrcp\") pod \"managed-serviceaccount-addon-agent-858bd75c58-j5tgk\" (UID: \"a16d32ce-a015-47d1-bc1f-09e61cff8f77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.424216 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.424202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2dj\" (UniqueName: \"kubernetes.io/projected/cc47e803-bc15-49c7-9d65-2ad5ca5cd890-kube-api-access-lx2dj\") pod \"migrator-6865c9967-h2drb\" (UID: \"cc47e803-bc15-49c7-9d65-2ad5ca5cd890\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"
Jul 28 12:36:13.426778 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.426761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a16d32ce-a015-47d1-bc1f-09e61cff8f77-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-858bd75c58-j5tgk\" (UID: \"a16d32ce-a015-47d1-bc1f-09e61cff8f77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.453874 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.453854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrcp\" (UniqueName: \"kubernetes.io/projected/a16d32ce-a015-47d1-bc1f-09e61cff8f77-kube-api-access-tbrcp\") pod \"managed-serviceaccount-addon-agent-858bd75c58-j5tgk\" (UID: \"a16d32ce-a015-47d1-bc1f-09e61cff8f77\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.525000 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.524931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2dj\" (UniqueName: \"kubernetes.io/projected/cc47e803-bc15-49c7-9d65-2ad5ca5cd890-kube-api-access-lx2dj\") pod \"migrator-6865c9967-h2drb\" (UID: \"cc47e803-bc15-49c7-9d65-2ad5ca5cd890\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"
Jul 28 12:36:13.534815 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:13.534796    2563 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found
Jul 28 12:36:13.534903 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:13.534815    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lx2dj for pod openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb: configmap "openshift-service-ca.crt" not found
Jul 28 12:36:13.534903 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:36:13.534889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc47e803-bc15-49c7-9d65-2ad5ca5cd890-kube-api-access-lx2dj podName:cc47e803-bc15-49c7-9d65-2ad5ca5cd890 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:14.034873348 +0000 UTC m=+82.253226038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lx2dj" (UniqueName: "kubernetes.io/projected/cc47e803-bc15-49c7-9d65-2ad5ca5cd890-kube-api-access-lx2dj") pod "migrator-6865c9967-h2drb" (UID: "cc47e803-bc15-49c7-9d65-2ad5ca5cd890") : configmap "openshift-service-ca.crt" not found
Jul 28 12:36:13.553593 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.553571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"
Jul 28 12:36:13.697537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.697499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk"]
Jul 28 12:36:13.701248 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:13.701210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda16d32ce_a015_47d1_bc1f_09e61cff8f77.slice/crio-24643593dd10e458d5b5f162361d2aaf615bf4b7fb0d4b27b77d3e9271e927b0 WatchSource:0}: Error finding container 24643593dd10e458d5b5f162361d2aaf615bf4b7fb0d4b27b77d3e9271e927b0: Status 404 returned error can't find the container with id 24643593dd10e458d5b5f162361d2aaf615bf4b7fb0d4b27b77d3e9271e927b0
Jul 28 12:36:13.733357 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.733325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk" event={"ID":"a16d32ce-a015-47d1-bc1f-09e61cff8f77","Type":"ContainerStarted","Data":"24643593dd10e458d5b5f162361d2aaf615bf4b7fb0d4b27b77d3e9271e927b0"}
Jul 28 12:36:13.888383 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.888302    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-jqxjb"]
Jul 28 12:36:13.890519 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.890503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:13.893145 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.893114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xxxh4\""
Jul 28 12:36:13.893263 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.893185    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:36:13.893263 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.893236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:36:13.909452 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.909426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-jqxjb"]
Jul 28 12:36:13.928241 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:13.928214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrf5\" (UniqueName: \"kubernetes.io/projected/1383c340-e6e9-4605-82eb-959f8b778bd7-kube-api-access-gnrf5\") pod \"downloads-7c97dc4555-jqxjb\" (UID: \"1383c340-e6e9-4605-82eb-959f8b778bd7\") " pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:14.028908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.028876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrf5\" (UniqueName: \"kubernetes.io/projected/1383c340-e6e9-4605-82eb-959f8b778bd7-kube-api-access-gnrf5\") pod \"downloads-7c97dc4555-jqxjb\" (UID: \"1383c340-e6e9-4605-82eb-959f8b778bd7\") " pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:14.040705 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.040676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrf5\" (UniqueName: \"kubernetes.io/projected/1383c340-e6e9-4605-82eb-959f8b778bd7-kube-api-access-gnrf5\") pod \"downloads-7c97dc4555-jqxjb\" (UID: \"1383c340-e6e9-4605-82eb-959f8b778bd7\") " pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:14.130143 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.130117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2dj\" (UniqueName: \"kubernetes.io/projected/cc47e803-bc15-49c7-9d65-2ad5ca5cd890-kube-api-access-lx2dj\") pod \"migrator-6865c9967-h2drb\" (UID: \"cc47e803-bc15-49c7-9d65-2ad5ca5cd890\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"
Jul 28 12:36:14.132568 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.132550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2dj\" (UniqueName: \"kubernetes.io/projected/cc47e803-bc15-49c7-9d65-2ad5ca5cd890-kube-api-access-lx2dj\") pod \"migrator-6865c9967-h2drb\" (UID: \"cc47e803-bc15-49c7-9d65-2ad5ca5cd890\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"
Jul 28 12:36:14.198979 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.198952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:14.235587 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.235549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"
Jul 28 12:36:14.344474 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.344449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-jqxjb"]
Jul 28 12:36:14.344714 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:14.344694    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1383c340_e6e9_4605_82eb_959f8b778bd7.slice/crio-513b9c3d58aaf8642f4d49708a9bf8db647fb364f399d80980a3fa5bb5861ece WatchSource:0}: Error finding container 513b9c3d58aaf8642f4d49708a9bf8db647fb364f399d80980a3fa5bb5861ece: Status 404 returned error can't find the container with id 513b9c3d58aaf8642f4d49708a9bf8db647fb364f399d80980a3fa5bb5861ece
Jul 28 12:36:14.380439 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.380412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb"]
Jul 28 12:36:14.383332 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:14.383306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc47e803_bc15_49c7_9d65_2ad5ca5cd890.slice/crio-84b4eaf95b65ade54729b799e440ba025f8ebeefc843fe1ffd74070356ab6d5c WatchSource:0}: Error finding container 84b4eaf95b65ade54729b799e440ba025f8ebeefc843fe1ffd74070356ab6d5c: Status 404 returned error can't find the container with id 84b4eaf95b65ade54729b799e440ba025f8ebeefc843fe1ffd74070356ab6d5c
Jul 28 12:36:14.736414 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.736379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb" event={"ID":"cc47e803-bc15-49c7-9d65-2ad5ca5cd890","Type":"ContainerStarted","Data":"84b4eaf95b65ade54729b799e440ba025f8ebeefc843fe1ffd74070356ab6d5c"}
Jul 28 12:36:14.737353 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:14.737329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-jqxjb" event={"ID":"1383c340-e6e9-4605-82eb-959f8b778bd7","Type":"ContainerStarted","Data":"513b9c3d58aaf8642f4d49708a9bf8db647fb364f399d80980a3fa5bb5861ece"}
Jul 28 12:36:15.744760 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:15.744726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb" event={"ID":"cc47e803-bc15-49c7-9d65-2ad5ca5cd890","Type":"ContainerStarted","Data":"ebffcf44b0cb3019f262431732534e9dac44afcd7971c360c39f7210c4cf89c7"}
Jul 28 12:36:15.745185 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:15.744791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb" event={"ID":"cc47e803-bc15-49c7-9d65-2ad5ca5cd890","Type":"ContainerStarted","Data":"41920b5a8ddcd6b484e46365e6002b188cc0ad7035d15222ed97877b01023a4f"}
Jul 28 12:36:15.765178 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:15.765127    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-h2drb" podStartSLOduration=1.616302626 podStartE2EDuration="2.76510759s" totalImagesPullingTime="1.148804964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:15.764152362 +0000 UTC m=+83.982505075" watchObservedRunningTime="2026-07-28 12:36:15.76510759 +0000 UTC m=+83.983460308"
Jul 28 12:36:19.269526 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:19.269487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:36:19.272581 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:19.272553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d14de7a2-2981-41a0-b3c2-980fd5ba681b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-q9znf\" (UID: \"d14de7a2-2981-41a0-b3c2-980fd5ba681b\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:36:19.515780 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:19.515751    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br6h9\""
Jul 28 12:36:19.523921 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:19.523861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"
Jul 28 12:36:19.653224 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:19.653191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf"]
Jul 28 12:36:19.656671 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:19.656638    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14de7a2_2981_41a0_b3c2_980fd5ba681b.slice/crio-6c4b4d82015f947d9d7717cd7a539faae6f399beddd9acbe2d502124a161c2a5 WatchSource:0}: Error finding container 6c4b4d82015f947d9d7717cd7a539faae6f399beddd9acbe2d502124a161c2a5: Status 404 returned error can't find the container with id 6c4b4d82015f947d9d7717cd7a539faae6f399beddd9acbe2d502124a161c2a5
Jul 28 12:36:19.760162 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:19.760118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf" event={"ID":"d14de7a2-2981-41a0-b3c2-980fd5ba681b","Type":"ContainerStarted","Data":"6c4b4d82015f947d9d7717cd7a539faae6f399beddd9acbe2d502124a161c2a5"}
Jul 28 12:36:21.769325 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:21.769283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf" event={"ID":"d14de7a2-2981-41a0-b3c2-980fd5ba681b","Type":"ContainerStarted","Data":"c8f0eb3b24697254ed7af852a66a23d6a599baeb7f7a8e2fb2494b84014f245a"}
Jul 28 12:36:23.780336 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:23.780297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk" event={"ID":"a16d32ce-a015-47d1-bc1f-09e61cff8f77","Type":"ContainerStarted","Data":"11459e9baa99b356d6a797284870f7b9ac5fc9114b8dae6f03042799de718e33"}
Jul 28 12:36:23.797279 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:23.797217    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-q9znf" podStartSLOduration=35.066997928 podStartE2EDuration="36.797199103s" totalImagesPullingTime="1.730201175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:21.788431421 +0000 UTC m=+90.006784134" watchObservedRunningTime="2026-07-28 12:36:23.797199103 +0000 UTC m=+92.015551823"
Jul 28 12:36:23.797667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:23.797608    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858bd75c58-j5tgk" podStartSLOduration=0.792560872 podStartE2EDuration="10.7975953s" totalImagesPullingTime="10.005034428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:23.797510655 +0000 UTC m=+92.015863345" watchObservedRunningTime="2026-07-28 12:36:23.7975953 +0000 UTC m=+92.015948014"
Jul 28 12:36:24.422940 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.422901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:24.425546 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.425521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb20ff7e-9864-4d09-b996-6a4f5e5c76ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bb42n\" (UID: \"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad\") " pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:24.673748 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.673659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bb42n"
Jul 28 12:36:24.840378 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.840350    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bb42n"]
Jul 28 12:36:24.844018 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.843992    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ff68b95fc-l829s"]
Jul 28 12:36:24.844491 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:24.844464    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb20ff7e_9864_4d09_b996_6a4f5e5c76ad.slice/crio-0840412bda22de198f38a31c8871b16f093b98313ca5fe5922cf80abfda955af WatchSource:0}: Error finding container 0840412bda22de198f38a31c8871b16f093b98313ca5fe5922cf80abfda955af: Status 404 returned error can't find the container with id 0840412bda22de198f38a31c8871b16f093b98313ca5fe5922cf80abfda955af
Jul 28 12:36:24.849491 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.849469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.852249 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.852227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:36:24.852361 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.852231    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:36:24.853414 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.853396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zh7zq\""
Jul 28 12:36:24.853753 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.853733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:36:24.853937 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.853794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:36:24.857918 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.857723    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:36:24.859313 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.859289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:36:24.872014 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.871995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff68b95fc-l829s"]
Jul 28 12:36:24.931804 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.931733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-config\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.931804 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.931790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-service-ca\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.931988 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.931888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-oauth-serving-cert\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.931988 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.931930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-oauth-config\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.931988 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.931969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-serving-cert\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.932135 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.931997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2xq\" (UniqueName: \"kubernetes.io/projected/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-kube-api-access-jh2xq\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:24.932135 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:24.932055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-trusted-ca-bundle\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.032745 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-config\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.032922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-service-ca\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.032922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-oauth-serving-cert\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.032922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-oauth-config\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.032922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-serving-cert\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.032922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2xq\" (UniqueName: \"kubernetes.io/projected/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-kube-api-access-jh2xq\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.033172 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.032939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-trusted-ca-bundle\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.033693 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.033585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-config\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.033810 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.033766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-service-ca\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.034058 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.034036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-oauth-serving-cert\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.034657 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.034637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-trusted-ca-bundle\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.035730 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.035709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-oauth-config\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.035893 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.035877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-serving-cert\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.041878 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.041854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2xq\" (UniqueName: \"kubernetes.io/projected/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-kube-api-access-jh2xq\") pod \"console-7ff68b95fc-l829s\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") " pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.160511 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.160478    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:25.344421 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.344349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff68b95fc-l829s"]
Jul 28 12:36:25.791812 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.791778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff68b95fc-l829s" event={"ID":"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e","Type":"ContainerStarted","Data":"ee7591edf11b9ade403b46a528aa0b3d980f5dd0a30ce56cebfee21ee923d8b4"}
Jul 28 12:36:25.793262 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:25.793189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bb42n" event={"ID":"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad","Type":"ContainerStarted","Data":"0840412bda22de198f38a31c8871b16f093b98313ca5fe5922cf80abfda955af"}
Jul 28 12:36:27.972492 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.972452    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"]
Jul 28 12:36:27.975936 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.975908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:27.980097 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.980066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lr5cs\""
Jul 28 12:36:27.980238 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.980131    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 12:36:27.980238 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.980074    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:36:27.980340 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.980074    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 12:36:27.992692 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:27.992665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"]
Jul 28 12:36:28.058396 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.058360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclql\" (UniqueName: \"kubernetes.io/projected/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-kube-api-access-vclql\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.058585 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.058432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.058641 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.058599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.058692 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.058640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.159705 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.159672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.159904 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.159752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.159904 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.159785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.159904 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.159810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclql\" (UniqueName: \"kubernetes.io/projected/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-kube-api-access-vclql\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.160399 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.160369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.162589 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.162565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.162755 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.162731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.170981 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.170959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclql\" (UniqueName: \"kubernetes.io/projected/a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4-kube-api-access-vclql\") pod \"prometheus-operator-84d69f794d-rt5m6\" (UID: \"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:28.288272 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:28.288200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"
Jul 28 12:36:32.096481 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.096442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:36:32.096981 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.096501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:36:32.099136 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.099114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9f34d2d2-3bbb-4f64-95e4-e2087909f7cd-metrics-tls\") pod \"dns-default-clr4g\" (UID: \"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd\") " pod="openshift-dns/dns-default-clr4g"
Jul 28 12:36:32.099343 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.099320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"image-registry-7698788d7d-6chn2\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") " pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:36:32.341609 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.341576    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w2rc8\""
Jul 28 12:36:32.350020 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.349947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:36:32.362239 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.362220    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l7wd\""
Jul 28 12:36:32.370619 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:32.370599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clr4g"
Jul 28 12:36:33.710079 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:33.710048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5tsz9"
Jul 28 12:36:35.232636 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:35.232605    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7698788d7d-6chn2"]
Jul 28 12:36:38.718260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.718119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7698788d7d-6chn2"]
Jul 28 12:36:38.722949 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:38.722916    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217ed5d2_02b2_4ca3_95a0_50d207754e72.slice/crio-02dea075c37a91a621583b4a607a377a41750166d1fdcdf9ae995f094ebc2c6a WatchSource:0}: Error finding container 02dea075c37a91a621583b4a607a377a41750166d1fdcdf9ae995f094ebc2c6a: Status 404 returned error can't find the container with id 02dea075c37a91a621583b4a607a377a41750166d1fdcdf9ae995f094ebc2c6a
Jul 28 12:36:38.732438 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.732410    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rt5m6"]
Jul 28 12:36:38.735361 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.735338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clr4g"]
Jul 28 12:36:38.735549 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:38.735524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ac5e55_bd18_4a6f_aaa1_2fd7811034b4.slice/crio-14befd14a01959499576a6687093049f2755cb74caec3dd66be863b02d46625e WatchSource:0}: Error finding container 14befd14a01959499576a6687093049f2755cb74caec3dd66be863b02d46625e: Status 404 returned error can't find the container with id 14befd14a01959499576a6687093049f2755cb74caec3dd66be863b02d46625e
Jul 28 12:36:38.739075 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:38.739050    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f34d2d2_3bbb_4f64_95e4_e2087909f7cd.slice/crio-f2e441e434c1e8f20b3915133897ac1a150402fdf51dfef2e0f7848109d3912a WatchSource:0}: Error finding container f2e441e434c1e8f20b3915133897ac1a150402fdf51dfef2e0f7848109d3912a: Status 404 returned error can't find the container with id f2e441e434c1e8f20b3915133897ac1a150402fdf51dfef2e0f7848109d3912a
Jul 28 12:36:38.841019 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.840943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clr4g" event={"ID":"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd","Type":"ContainerStarted","Data":"f2e441e434c1e8f20b3915133897ac1a150402fdf51dfef2e0f7848109d3912a"}
Jul 28 12:36:38.843127 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.842863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7698788d7d-6chn2" event={"ID":"217ed5d2-02b2-4ca3-95a0-50d207754e72","Type":"ContainerStarted","Data":"e941e37b68584b73f68f45412cd31b3e22480a429afbdba216c3fe56640c5d99"}
Jul 28 12:36:38.843127 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.842917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7698788d7d-6chn2" event={"ID":"217ed5d2-02b2-4ca3-95a0-50d207754e72","Type":"ContainerStarted","Data":"02dea075c37a91a621583b4a607a377a41750166d1fdcdf9ae995f094ebc2c6a"}
Jul 28 12:36:38.843127 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.843055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:36:38.846068 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.846038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff68b95fc-l829s" event={"ID":"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e","Type":"ContainerStarted","Data":"8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a"}
Jul 28 12:36:38.848330 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.848187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bb42n" event={"ID":"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad","Type":"ContainerStarted","Data":"c7bee0eb25d8035918e6364ec3218633b364b08ba6a4761a5ce17464afe459f3"}
Jul 28 12:36:38.849962 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.849938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6" event={"ID":"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4","Type":"ContainerStarted","Data":"14befd14a01959499576a6687093049f2755cb74caec3dd66be863b02d46625e"}
Jul 28 12:36:38.851744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.851719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-jqxjb" event={"ID":"1383c340-e6e9-4605-82eb-959f8b778bd7","Type":"ContainerStarted","Data":"5a880862b03cb8f171f82ecd05f2930b367fc31cbb0bdc23d36be15ca9408476"}
Jul 28 12:36:38.852121 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.852069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:38.860003 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.859974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-jqxjb"
Jul 28 12:36:38.864846 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.864719    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7698788d7d-6chn2" podStartSLOduration=106.86470582 podStartE2EDuration="1m46.86470582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:38.86411105 +0000 UTC m=+107.082463761" watchObservedRunningTime="2026-07-28 12:36:38.86470582 +0000 UTC m=+107.083058532"
Jul 28 12:36:38.910636 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.910580    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ff68b95fc-l829s" podStartSLOduration=1.727263327 podStartE2EDuration="14.910564635s" totalImagesPullingTime="13.183301308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:38.910013618 +0000 UTC m=+107.128366343" watchObservedRunningTime="2026-07-28 12:36:38.910564635 +0000 UTC m=+107.128917348"
Jul 28 12:36:38.911617 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:38.911571    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-jqxjb" podStartSLOduration=1.72708047 podStartE2EDuration="25.911558404s" totalImagesPullingTime="24.184477934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:38.884138985 +0000 UTC m=+107.102491696" watchObservedRunningTime="2026-07-28 12:36:38.911558404 +0000 UTC m=+107.129911119"
Jul 28 12:36:45.161011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.160977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:45.161457 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.161057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:45.166697 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.166675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:45.887676 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.887637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6" event={"ID":"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4","Type":"ContainerStarted","Data":"a83c3ae22c21ace9a57becbc4cd7c0251932208046ce3d18024a19c349b69399"}
Jul 28 12:36:45.887906 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.887705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6" event={"ID":"a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4","Type":"ContainerStarted","Data":"20a56a088b0df89f1f0b9585835182fae720b977aad5b4de70da21e951f350bd"}
Jul 28 12:36:45.889839 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.889798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clr4g" event={"ID":"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd","Type":"ContainerStarted","Data":"f383c160fbe9e72b11b4bde36342e1d502dd540a87fe1016e1f9fa8dea047f54"}
Jul 28 12:36:45.889961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.889865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clr4g" event={"ID":"9f34d2d2-3bbb-4f64-95e4-e2087909f7cd","Type":"ContainerStarted","Data":"f75864c802b32f404f6c159d34aeea7459e71634748004c5164917921fab4a58"}
Jul 28 12:36:45.890022 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.889950    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-clr4g"
Jul 28 12:36:45.891749 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.891722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bb42n" event={"ID":"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad","Type":"ContainerStarted","Data":"ffaefe6ccf8299388fe3c050502e6ab888cc43406bea63a1b546b628a24ee53d"}
Jul 28 12:36:45.896339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.896319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:36:45.907312 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.907266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-rt5m6" podStartSLOduration=12.715057765 podStartE2EDuration="18.90725111s" totalImagesPullingTime="6.192193345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:45.905856183 +0000 UTC m=+114.124208897" watchObservedRunningTime="2026-07-28 12:36:45.90725111 +0000 UTC m=+114.125603823"
Jul 28 12:36:45.924484 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:45.924441    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-clr4g" podStartSLOduration=71.735553412 podStartE2EDuration="1m17.92442435s" totalImagesPullingTime="6.188870938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:45.923004534 +0000 UTC m=+114.141357260" watchObservedRunningTime="2026-07-28 12:36:45.92442435 +0000 UTC m=+114.142777062"
Jul 28 12:36:47.400836 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.400793    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qtxlb"]
Jul 28 12:36:47.426243 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.426214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.430627 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.430420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:36:47.432965 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.432944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r62cm\""
Jul 28 12:36:47.433140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.433123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:36:47.433289 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.433266    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:36:47.437224 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-tls\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437494 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-root\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437494 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-sys\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437494 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kb6\" (UniqueName: \"kubernetes.io/projected/42b85603-415a-425d-a57f-85fdcbb354de-kube-api-access-68kb6\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437494 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437780 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42b85603-415a-425d-a57f-85fdcbb354de-metrics-client-ca\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437780 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437780 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-textfile\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.437780 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.437750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-wtmp\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539132 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-root\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-sys\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kb6\" (UniqueName: \"kubernetes.io/projected/42b85603-415a-425d-a57f-85fdcbb354de-kube-api-access-68kb6\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42b85603-415a-425d-a57f-85fdcbb354de-metrics-client-ca\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-textfile\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-wtmp\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-tls\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.539756 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-root\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.540706 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.539310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-sys\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.540706 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.540116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-textfile\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.540706 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.540443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42b85603-415a-425d-a57f-85fdcbb354de-metrics-client-ca\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.540706 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.540574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.540706 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.540584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-wtmp\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.543228 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.543184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-tls\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.550231 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.550173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42b85603-415a-425d-a57f-85fdcbb354de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.553766 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.553723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kb6\" (UniqueName: \"kubernetes.io/projected/42b85603-415a-425d-a57f-85fdcbb354de-kube-api-access-68kb6\") pod \"node-exporter-qtxlb\" (UID: \"42b85603-415a-425d-a57f-85fdcbb354de\") " pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:47.741161 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:47.741122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qtxlb"
Jul 28 12:36:48.418058 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.418023    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:48.518437 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.518404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:48.518608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.518576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.521512 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:36:48.521512 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521380    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:36:48.521512 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:36:48.521690 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:36:48.521690 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521651    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4gt2q\""
Jul 28 12:36:48.521690 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:36:48.521934 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.521917    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:36:48.522274 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.522094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:36:48.522274 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.522134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:36:48.522274 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.522163    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-out\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-web-config\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.547994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.548035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.548127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.548164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.550202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.548187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9l8g\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-kube-api-access-t9l8g\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.650757 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.650952 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.650952 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-out\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.650952 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-web-config\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.650952 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.650952 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.650995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.651020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.651057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.651079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.651101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9l8g\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-kube-api-access-t9l8g\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651234 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.651224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.651998 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.651777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.652602 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.652274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.655717 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.655428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.655717 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.655633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-out\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.655717 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.655675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.656065 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.656041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.657301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.657272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.657719 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.657699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.657803 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.657750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-web-config\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.658398 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.658380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.658398 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.658379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.661326 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.661308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9l8g\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-kube-api-access-t9l8g\") pod \"alertmanager-main-0\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:48.831146 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:48.831105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:49.637138 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:49.637104    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b85603_415a_425d_a57f_85fdcbb354de.slice/crio-54759d1b7d90d6073c8a42e5948874d13c3ea70480025e31e3552c4398f65389 WatchSource:0}: Error finding container 54759d1b7d90d6073c8a42e5948874d13c3ea70480025e31e3552c4398f65389: Status 404 returned error can't find the container with id 54759d1b7d90d6073c8a42e5948874d13c3ea70480025e31e3552c4398f65389
Jul 28 12:36:49.765655 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:49.765629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:49.766951 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:49.766925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3db186b_39a7_4c34_b0e6_c68b59ed6621.slice/crio-36a17b175fe5cb05def6dc374fe349ceed353f2008ecb64b9e01ce5997602958 WatchSource:0}: Error finding container 36a17b175fe5cb05def6dc374fe349ceed353f2008ecb64b9e01ce5997602958: Status 404 returned error can't find the container with id 36a17b175fe5cb05def6dc374fe349ceed353f2008ecb64b9e01ce5997602958
Jul 28 12:36:49.919607 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:49.919532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtxlb" event={"ID":"42b85603-415a-425d-a57f-85fdcbb354de","Type":"ContainerStarted","Data":"54759d1b7d90d6073c8a42e5948874d13c3ea70480025e31e3552c4398f65389"}
Jul 28 12:36:49.920681 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:49.920652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"36a17b175fe5cb05def6dc374fe349ceed353f2008ecb64b9e01ce5997602958"}
Jul 28 12:36:50.926660 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:50.926621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bb42n" event={"ID":"cb20ff7e-9864-4d09-b996-6a4f5e5c76ad","Type":"ContainerStarted","Data":"8676a396ec28e79d8b1ffd06219b2f7984203dcd0e522bf7d1ee1890772512c6"}
Jul 28 12:36:50.950597 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:50.950532    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bb42n" podStartSLOduration=34.04076228 podStartE2EDuration="58.950507263s" totalImagesPullingTime="24.909744983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:50.948128426 +0000 UTC m=+119.166481178" watchObservedRunningTime="2026-07-28 12:36:50.950507263 +0000 UTC m=+119.168859975"
Jul 28 12:36:51.774541 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.774501    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5694649-m945p"]
Jul 28 12:36:51.783140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.783113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.785786 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.785760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 12:36:51.786713 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.786691    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 12:36:51.786852 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.786787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-973fgeg6q2g1a\""
Jul 28 12:36:51.786954 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.786934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mbb8x\""
Jul 28 12:36:51.786954 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.786950    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 12:36:51.787107 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.786934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 12:36:51.790288 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.790227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5694649-m945p"]
Jul 28 12:36:51.878646 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7cc71eb8-0f26-446b-9581-6f801391c2c1-metrics-server-audit-profiles\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.878852 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-client-ca-bundle\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.878852 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7cc71eb8-0f26-446b-9581-6f801391c2c1-audit-log\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.878852 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-secret-metrics-server-tls\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.878852 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-secret-metrics-server-client-certs\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.879088 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc71eb8-0f26-446b-9581-6f801391c2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.879088 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.878957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5tv\" (UniqueName: \"kubernetes.io/projected/7cc71eb8-0f26-446b-9581-6f801391c2c1-kube-api-access-bs5tv\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.932591 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.932555    2563 generic.go:356] "Generic (PLEG): container finished" podID="42b85603-415a-425d-a57f-85fdcbb354de" containerID="75b0cbdc72e12ea1e5fc35f46f98a21f050f7ecdd0cd8b9500ee83851d23444b" exitCode=0
Jul 28 12:36:51.933053 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.932633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtxlb" event={"ID":"42b85603-415a-425d-a57f-85fdcbb354de","Type":"ContainerDied","Data":"75b0cbdc72e12ea1e5fc35f46f98a21f050f7ecdd0cd8b9500ee83851d23444b"}
Jul 28 12:36:51.980409 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-secret-metrics-server-tls\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.980409 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-secret-metrics-server-client-certs\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.980409 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc71eb8-0f26-446b-9581-6f801391c2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.980657 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5tv\" (UniqueName: \"kubernetes.io/projected/7cc71eb8-0f26-446b-9581-6f801391c2c1-kube-api-access-bs5tv\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.980734 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7cc71eb8-0f26-446b-9581-6f801391c2c1-metrics-server-audit-profiles\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.980810 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-client-ca-bundle\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.980931 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.980909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7cc71eb8-0f26-446b-9581-6f801391c2c1-audit-log\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.981242 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.981219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc71eb8-0f26-446b-9581-6f801391c2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.981447 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.981306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7cc71eb8-0f26-446b-9581-6f801391c2c1-audit-log\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.982099 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.982074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7cc71eb8-0f26-446b-9581-6f801391c2c1-metrics-server-audit-profiles\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.983266 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.983244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-secret-metrics-server-tls\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.983372 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.983290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-client-ca-bundle\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.983372 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.983305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7cc71eb8-0f26-446b-9581-6f801391c2c1-secret-metrics-server-client-certs\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:51.988511 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:51.988488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5tv\" (UniqueName: \"kubernetes.io/projected/7cc71eb8-0f26-446b-9581-6f801391c2c1-kube-api-access-bs5tv\") pod \"metrics-server-5694649-m945p\" (UID: \"7cc71eb8-0f26-446b-9581-6f801391c2c1\") " pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:52.093664 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.093638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:36:52.260858 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.260815    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5694649-m945p"]
Jul 28 12:36:52.396259 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:52.396223    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc71eb8_0f26_446b_9581_6f801391c2c1.slice/crio-33761d71706e82b71770a196e279e53516a0be46d0a94d2ecbfd71558704093e WatchSource:0}: Error finding container 33761d71706e82b71770a196e279e53516a0be46d0a94d2ecbfd71558704093e: Status 404 returned error can't find the container with id 33761d71706e82b71770a196e279e53516a0be46d0a94d2ecbfd71558704093e
Jul 28 12:36:52.938322 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.938235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtxlb" event={"ID":"42b85603-415a-425d-a57f-85fdcbb354de","Type":"ContainerStarted","Data":"f404b5431b969d22a14aac283ca268cde2d1e9dd3b21148e9ebcac2a093ee938"}
Jul 28 12:36:52.938322 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.938295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtxlb" event={"ID":"42b85603-415a-425d-a57f-85fdcbb354de","Type":"ContainerStarted","Data":"d923fd3306a4ad9f11efc9c264d54863eca0ee007783068a08c7215b79e58218"}
Jul 28 12:36:52.939456 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.939432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5694649-m945p" event={"ID":"7cc71eb8-0f26-446b-9581-6f801391c2c1","Type":"ContainerStarted","Data":"33761d71706e82b71770a196e279e53516a0be46d0a94d2ecbfd71558704093e"}
Jul 28 12:36:52.940746 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.940715    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d" exitCode=0
Jul 28 12:36:52.940882 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.940780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"}
Jul 28 12:36:52.957780 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:52.957736    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qtxlb" podStartSLOduration=4.030866168 podStartE2EDuration="5.957721468s" totalImagesPullingTime="1.815196367s" totalInitContainerRuntime="111.658933ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:52.956788964 +0000 UTC m=+121.175141677" watchObservedRunningTime="2026-07-28 12:36:52.957721468 +0000 UTC m=+121.176074180"
Jul 28 12:36:55.904514 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.904484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-clr4g"
Jul 28 12:36:55.955582 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.955539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5694649-m945p" event={"ID":"7cc71eb8-0f26-446b-9581-6f801391c2c1","Type":"ContainerStarted","Data":"9e7ef8d4f4b36c754afae87e03e8a5ec153033e36abd3bbfec64702cb6b5f3c4"}
Jul 28 12:36:55.957007 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.956979    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed5a6cde-92cb-49ff-bba3-41f0a7e897fd" containerID="6ebc513adb6885b5b4e3f36d28686aaa6c254b3389892f240995e1c2050c6698" exitCode=0
Jul 28 12:36:55.957139 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.957046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z" event={"ID":"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd","Type":"ContainerDied","Data":"6ebc513adb6885b5b4e3f36d28686aaa6c254b3389892f240995e1c2050c6698"}
Jul 28 12:36:55.957363 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.957344    2563 scope.go:117] "RemoveContainer" containerID="6ebc513adb6885b5b4e3f36d28686aaa6c254b3389892f240995e1c2050c6698"
Jul 28 12:36:55.960976 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.960957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"}
Jul 28 12:36:55.961074 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.961005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"}
Jul 28 12:36:55.961074 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.961029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"}
Jul 28 12:36:55.961074 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.961054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"}
Jul 28 12:36:55.961173 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.961077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"}
Jul 28 12:36:55.975597 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:55.975548    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5694649-m945p" podStartSLOduration=2.517762964 podStartE2EDuration="4.975534901s" totalImagesPullingTime="2.457771937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:55.974194514 +0000 UTC m=+124.192547230" watchObservedRunningTime="2026-07-28 12:36:55.975534901 +0000 UTC m=+124.193887613"
Jul 28 12:36:56.967595 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:56.967560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lld9z" event={"ID":"ed5a6cde-92cb-49ff-bba3-41f0a7e897fd","Type":"ContainerStarted","Data":"0f32f8adf91a4b8d5020d77678d6253baa658306668d58da3d7b3be74b7a3543"}
Jul 28 12:36:56.971662 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:56.971634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerStarted","Data":"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"}
Jul 28 12:36:57.025157 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.024282    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.140502304 podStartE2EDuration="9.024257954s" totalImagesPullingTime="5.863226906s" totalInitContainerRuntime="20.528744ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:57.022189524 +0000 UTC m=+125.240542236" watchObservedRunningTime="2026-07-28 12:36:57.024257954 +0000 UTC m=+125.242610667"
Jul 28 12:36:57.873549 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.873517    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6779bb5-lm4rr"]
Jul 28 12:36:57.895448 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.895421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6779bb5-lm4rr"]
Jul 28 12:36:57.895627 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.895520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935719 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-trusted-ca-bundle\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935719 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8b5l\" (UniqueName: \"kubernetes.io/projected/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-kube-api-access-p8b5l\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-service-ca\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-config\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-serving-cert\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-oauth-config\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:57.935961 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:57.935940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-oauth-serving-cert\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.036705 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.036669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-oauth-config\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.036705 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.036710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-oauth-serving-cert\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.036930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-trusted-ca-bundle\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.036994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8b5l\" (UniqueName: \"kubernetes.io/projected/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-kube-api-access-p8b5l\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-service-ca\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-config\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-serving-cert\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037740 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-config\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.037956 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-service-ca\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.038034 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-trusted-ca-bundle\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.038034 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.037978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-oauth-serving-cert\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.040101 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.040075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-serving-cert\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.040588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.040569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-oauth-config\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.047903 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.047882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8b5l\" (UniqueName: \"kubernetes.io/projected/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-kube-api-access-p8b5l\") pod \"console-5f6779bb5-lm4rr\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") " pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.207550 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.207521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:36:58.383923 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.383888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6779bb5-lm4rr"]
Jul 28 12:36:58.386918 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:36:58.386875    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cdf838_5722_443e_a5c9_fbc6bd2ca6dd.slice/crio-286936e82f60e71f1e239c1d81b66fb9814f70f308c2c7664b97f435a677f6c1 WatchSource:0}: Error finding container 286936e82f60e71f1e239c1d81b66fb9814f70f308c2c7664b97f435a677f6c1: Status 404 returned error can't find the container with id 286936e82f60e71f1e239c1d81b66fb9814f70f308c2c7664b97f435a677f6c1
Jul 28 12:36:58.849031 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.848967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:36:58.982772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.982733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6779bb5-lm4rr" event={"ID":"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd","Type":"ContainerStarted","Data":"a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39"}
Jul 28 12:36:58.982950 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:58.982780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6779bb5-lm4rr" event={"ID":"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd","Type":"ContainerStarted","Data":"286936e82f60e71f1e239c1d81b66fb9814f70f308c2c7664b97f435a677f6c1"}
Jul 28 12:36:59.002064 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:36:59.002019    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f6779bb5-lm4rr" podStartSLOduration=2.002000211 podStartE2EDuration="2.002000211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:59.000707987 +0000 UTC m=+127.219060736" watchObservedRunningTime="2026-07-28 12:36:59.002000211 +0000 UTC m=+127.220352924"
Jul 28 12:37:01.164725 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:01.164681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:37:01.167151 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:01.167127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae2f41f-73c2-451c-b9fe-c6a94c59a238-metrics-certs\") pod \"network-metrics-daemon-dqk86\" (UID: \"7ae2f41f-73c2-451c-b9fe-c6a94c59a238\") " pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:37:01.455182 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:01.455155    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kxg2r\""
Jul 28 12:37:01.463276 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:01.463262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqk86"
Jul 28 12:37:01.618763 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:01.618734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqk86"]
Jul 28 12:37:01.622235 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:37:01.622206    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae2f41f_73c2_451c_b9fe_c6a94c59a238.slice/crio-1a14e585d3cae597a19f8a253f1162054c83c84016b2b5389c754087922ed29f WatchSource:0}: Error finding container 1a14e585d3cae597a19f8a253f1162054c83c84016b2b5389c754087922ed29f: Status 404 returned error can't find the container with id 1a14e585d3cae597a19f8a253f1162054c83c84016b2b5389c754087922ed29f
Jul 28 12:37:01.994518 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:01.994483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqk86" event={"ID":"7ae2f41f-73c2-451c-b9fe-c6a94c59a238","Type":"ContainerStarted","Data":"1a14e585d3cae597a19f8a253f1162054c83c84016b2b5389c754087922ed29f"}
Jul 28 12:37:03.865514 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:03.865471    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7698788d7d-6chn2" podUID="217ed5d2-02b2-4ca3-95a0-50d207754e72" containerName="registry" containerID="cri-o://e941e37b68584b73f68f45412cd31b3e22480a429afbdba216c3fe56640c5d99" gracePeriod=30
Jul 28 12:37:04.004042 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.004012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqk86" event={"ID":"7ae2f41f-73c2-451c-b9fe-c6a94c59a238","Type":"ContainerStarted","Data":"fd685f095c8b84572c882598ba4dc02af255b5e0337058c293b58a7f9bbc2f50"}
Jul 28 12:37:04.004185 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.004078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqk86" event={"ID":"7ae2f41f-73c2-451c-b9fe-c6a94c59a238","Type":"ContainerStarted","Data":"d0c1292af82626504d99ed59d31c543d77a2837231ec45e93c949b2a2375a656"}
Jul 28 12:37:04.005703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.005677    2563 generic.go:356] "Generic (PLEG): container finished" podID="217ed5d2-02b2-4ca3-95a0-50d207754e72" containerID="e941e37b68584b73f68f45412cd31b3e22480a429afbdba216c3fe56640c5d99" exitCode=0
Jul 28 12:37:04.005808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.005754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7698788d7d-6chn2" event={"ID":"217ed5d2-02b2-4ca3-95a0-50d207754e72","Type":"ContainerDied","Data":"e941e37b68584b73f68f45412cd31b3e22480a429afbdba216c3fe56640c5d99"}
Jul 28 12:37:04.024364 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.024316    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dqk86" podStartSLOduration=130.283168083 podStartE2EDuration="2m12.024302075s" totalImagesPullingTime="1.741133992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:37:04.022553256 +0000 UTC m=+132.240906013" watchObservedRunningTime="2026-07-28 12:37:04.024302075 +0000 UTC m=+132.242654787"
Jul 28 12:37:04.146280 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.146213    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:37:04.190924 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.190895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-trusted-ca\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191063 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.190933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-certificates\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191063 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.190975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gxbm\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-kube-api-access-7gxbm\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191063 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/217ed5d2-02b2-4ca3-95a0-50d207754e72-ca-trust-extracted\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-image-registry-private-configuration\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-installation-pull-secrets\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-bound-sa-token\") pod \"217ed5d2-02b2-4ca3-95a0-50d207754e72\" (UID: \"217ed5d2-02b2-4ca3-95a0-50d207754e72\") "
Jul 28 12:37:04.191537 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:04.191608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.191552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:04.193660 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.193630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:04.193773 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.193727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:04.193891 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.193873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-kube-api-access-7gxbm" (OuterVolumeSpecName: "kube-api-access-7gxbm") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "kube-api-access-7gxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:04.194091 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.194071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:04.194189 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.194099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:04.199514 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.199484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217ed5d2-02b2-4ca3-95a0-50d207754e72-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "217ed5d2-02b2-4ca3-95a0-50d207754e72" (UID: "217ed5d2-02b2-4ca3-95a0-50d207754e72"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:37:04.292087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292059    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-bound-sa-token\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292084    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-trusted-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292094    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-certificates\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292103    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxbm\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-kube-api-access-7gxbm\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292112    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/217ed5d2-02b2-4ca3-95a0-50d207754e72-ca-trust-extracted\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292122    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/217ed5d2-02b2-4ca3-95a0-50d207754e72-registry-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292130    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-image-registry-private-configuration\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:04.292301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:04.292140    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/217ed5d2-02b2-4ca3-95a0-50d207754e72-installation-pull-secrets\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:05.012604 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:05.012573    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7698788d7d-6chn2"
Jul 28 12:37:05.013177 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:05.012618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7698788d7d-6chn2" event={"ID":"217ed5d2-02b2-4ca3-95a0-50d207754e72","Type":"ContainerDied","Data":"02dea075c37a91a621583b4a607a377a41750166d1fdcdf9ae995f094ebc2c6a"}
Jul 28 12:37:05.013177 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:05.012680    2563 scope.go:117] "RemoveContainer" containerID="e941e37b68584b73f68f45412cd31b3e22480a429afbdba216c3fe56640c5d99"
Jul 28 12:37:05.029302 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:05.029280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7698788d7d-6chn2"]
Jul 28 12:37:05.034877 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:05.034856    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7698788d7d-6chn2"]
Jul 28 12:37:06.343885 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:06.343851    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217ed5d2-02b2-4ca3-95a0-50d207754e72" path="/var/lib/kubelet/pods/217ed5d2-02b2-4ca3-95a0-50d207754e72/volumes"
Jul 28 12:37:08.207889 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:08.207856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:37:08.208259 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:08.207907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:37:08.212366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:08.212347    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:37:09.032246 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:09.032218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:37:09.081353 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:09.081318    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff68b95fc-l829s"]
Jul 28 12:37:12.093973 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:12.093944    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:37:12.094366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:12.094021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:37:18.138462 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:18.138416    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jnrmn_18018de2-4df9-4e7b-bbe9-6aab7abf990a/serve-healthcheck-canary/0.log"
Jul 28 12:37:32.099530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:32.099502    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:37:32.103414 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:32.103391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5694649-m945p"
Jul 28 12:37:34.101928 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.101862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7ff68b95fc-l829s" podUID="8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" containerName="console" containerID="cri-o://8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a" gracePeriod=15
Jul 28 12:37:34.336194 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.336172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:37:34.350346 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh2xq\" (UniqueName: \"kubernetes.io/projected/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-kube-api-access-jh2xq\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350466 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-serving-cert\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350466 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-config\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350466 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-oauth-serving-cert\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350632 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-service-ca\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350632 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-oauth-config\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350632 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-trusted-ca-bundle\") pod \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\" (UID: \"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e\") "
Jul 28 12:37:34.350883 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-config" (OuterVolumeSpecName: "console-config") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:34.350883 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:34.351017 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.350887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:34.351185 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.351158    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:34.352673 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.352621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-kube-api-access-jh2xq" (OuterVolumeSpecName: "kube-api-access-jh2xq") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "kube-api-access-jh2xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:34.352673 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.352652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:34.352906 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.352881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" (UID: "8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:34.451357 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451325    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2xq\" (UniqueName: \"kubernetes.io/projected/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-kube-api-access-jh2xq\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:34.451357 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451355    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:34.451553 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451366    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:34.451553 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451375    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-oauth-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:34.451553 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451383    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-service-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:34.451553 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451391    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-console-oauth-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:34.451553 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:34.451398    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:37:35.132034 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.132002    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" containerID="8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a" exitCode=0
Jul 28 12:37:35.132472 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.132045    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff68b95fc-l829s"
Jul 28 12:37:35.132472 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.132085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff68b95fc-l829s" event={"ID":"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e","Type":"ContainerDied","Data":"8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a"}
Jul 28 12:37:35.132472 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.132140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff68b95fc-l829s" event={"ID":"8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e","Type":"ContainerDied","Data":"ee7591edf11b9ade403b46a528aa0b3d980f5dd0a30ce56cebfee21ee923d8b4"}
Jul 28 12:37:35.132472 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.132165    2563 scope.go:117] "RemoveContainer" containerID="8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a"
Jul 28 12:37:35.140649 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.140633    2563 scope.go:117] "RemoveContainer" containerID="8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a"
Jul 28 12:37:35.140937 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:37:35.140916    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a\": container with ID starting with 8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a not found: ID does not exist" containerID="8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a"
Jul 28 12:37:35.141005 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.140944    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a"} err="failed to get container status \"8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a\": rpc error: code = NotFound desc = could not find container \"8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a\": container with ID starting with 8ea679bdb8474f951e42b0192e441b391c2e409519ac595aed3990d25ad6f60a not found: ID does not exist"
Jul 28 12:37:35.154136 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.154096    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff68b95fc-l829s"]
Jul 28 12:37:35.158396 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:35.158377    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ff68b95fc-l829s"]
Jul 28 12:37:36.343388 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:37:36.343346    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" path="/var/lib/kubelet/pods/8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e/volumes"
Jul 28 12:38:07.730459 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:38:07.731011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730736    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="alertmanager" containerID="cri-o://20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4" gracePeriod=120
Jul 28 12:38:07.731011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730839    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-metric" containerID="cri-o://2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa" gracePeriod=120
Jul 28 12:38:07.731011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730894    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy" containerID="cri-o://59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff" gracePeriod=120
Jul 28 12:38:07.731011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730892    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-web" containerID="cri-o://ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9" gracePeriod=120
Jul 28 12:38:07.731011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730853    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="prom-label-proxy" containerID="cri-o://5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089" gracePeriod=120
Jul 28 12:38:07.731284 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:07.730906    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="config-reloader" containerID="cri-o://7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9" gracePeriod=120
Jul 28 12:38:08.258607 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258578    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089" exitCode=0
Jul 28 12:38:08.258607 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258603    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff" exitCode=0
Jul 28 12:38:08.258607 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258610    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9" exitCode=0
Jul 28 12:38:08.258607 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258616    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4" exitCode=0
Jul 28 12:38:08.258898 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"}
Jul 28 12:38:08.258898 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"}
Jul 28 12:38:08.258898 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"}
Jul 28 12:38:08.258898 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.258770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"}
Jul 28 12:38:08.970446 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:08.970414    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.106352 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-out\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106352 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-web-config\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106352 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-volume\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106615 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-main-tls\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106615 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-main-db\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106615 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106436    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-metrics-client-ca\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106615 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-tls-assets\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106615 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106900 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:38:09.106900 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106807    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-cluster-tls-config\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.106900 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:09.107139 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-web\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.107139 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.107139 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.106989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9l8g\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-kube-api-access-t9l8g\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.107139 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.107023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-trusted-ca-bundle\") pod \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\" (UID: \"c3db186b-39a7-4c34-b0e6-c68b59ed6621\") "
Jul 28 12:38:09.107449 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.107433    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-main-db\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.107510 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.107454    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-metrics-client-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.108134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.107880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:09.109479 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.109448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.110551 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.110528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-volume" (OuterVolumeSpecName: "config-volume") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.110650 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.110551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.110650 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.110573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-out" (OuterVolumeSpecName: "config-out") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:38:09.110772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.110642    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.110772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.110662    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:38:09.110929 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.110906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.111301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.111271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-kube-api-access-t9l8g" (OuterVolumeSpecName: "kube-api-access-t9l8g") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "kube-api-access-t9l8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:38:09.114584 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.114540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.120767 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.120746    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-web-config" (OuterVolumeSpecName: "web-config") pod "c3db186b-39a7-4c34-b0e6-c68b59ed6621" (UID: "c3db186b-39a7-4c34-b0e6-c68b59ed6621"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:09.208555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208530    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3db186b-39a7-4c34-b0e6-c68b59ed6621-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208555 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208554    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-out\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208564    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-web-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208572    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-config-volume\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208580    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-main-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208588    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-tls-assets\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208596    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208604    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-cluster-tls-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208612    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208621    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3db186b-39a7-4c34-b0e6-c68b59ed6621-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.208687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.208629    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9l8g\" (UniqueName: \"kubernetes.io/projected/c3db186b-39a7-4c34-b0e6-c68b59ed6621-kube-api-access-t9l8g\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:09.266088 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266057    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa" exitCode=0
Jul 28 12:38:09.266088 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266083    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerID="ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9" exitCode=0
Jul 28 12:38:09.266304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"}
Jul 28 12:38:09.266304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"}
Jul 28 12:38:09.266304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266144    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.266304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3db186b-39a7-4c34-b0e6-c68b59ed6621","Type":"ContainerDied","Data":"36a17b175fe5cb05def6dc374fe349ceed353f2008ecb64b9e01ce5997602958"}
Jul 28 12:38:09.266304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.266263    2563 scope.go:117] "RemoveContainer" containerID="5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"
Jul 28 12:38:09.273988 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.273966    2563 scope.go:117] "RemoveContainer" containerID="2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"
Jul 28 12:38:09.281648 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.281631    2563 scope.go:117] "RemoveContainer" containerID="59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"
Jul 28 12:38:09.288196 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.288179    2563 scope.go:117] "RemoveContainer" containerID="ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"
Jul 28 12:38:09.291644 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.291621    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:38:09.295873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.295856    2563 scope.go:117] "RemoveContainer" containerID="7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"
Jul 28 12:38:09.296793 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.296776    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:38:09.302628 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.302607    2563 scope.go:117] "RemoveContainer" containerID="20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"
Jul 28 12:38:09.309739 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.309723    2563 scope.go:117] "RemoveContainer" containerID="41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"
Jul 28 12:38:09.316526 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.316509    2563 scope.go:117] "RemoveContainer" containerID="5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"
Jul 28 12:38:09.316776 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.316760    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089\": container with ID starting with 5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089 not found: ID does not exist" containerID="5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"
Jul 28 12:38:09.316870 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.316786    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"} err="failed to get container status \"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089\": rpc error: code = NotFound desc = could not find container \"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089\": container with ID starting with 5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089 not found: ID does not exist"
Jul 28 12:38:09.316870 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.316813    2563 scope.go:117] "RemoveContainer" containerID="2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"
Jul 28 12:38:09.317114 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.317097    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa\": container with ID starting with 2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa not found: ID does not exist" containerID="2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"
Jul 28 12:38:09.317170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317119    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"} err="failed to get container status \"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa\": rpc error: code = NotFound desc = could not find container \"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa\": container with ID starting with 2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa not found: ID does not exist"
Jul 28 12:38:09.317170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317135    2563 scope.go:117] "RemoveContainer" containerID="59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"
Jul 28 12:38:09.317388 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.317369    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff\": container with ID starting with 59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff not found: ID does not exist" containerID="59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"
Jul 28 12:38:09.317431 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317393    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"} err="failed to get container status \"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff\": rpc error: code = NotFound desc = could not find container \"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff\": container with ID starting with 59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff not found: ID does not exist"
Jul 28 12:38:09.317431 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317408    2563 scope.go:117] "RemoveContainer" containerID="ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"
Jul 28 12:38:09.317629 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.317613    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9\": container with ID starting with ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9 not found: ID does not exist" containerID="ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"
Jul 28 12:38:09.317689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317636    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"} err="failed to get container status \"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9\": rpc error: code = NotFound desc = could not find container \"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9\": container with ID starting with ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9 not found: ID does not exist"
Jul 28 12:38:09.317689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317660    2563 scope.go:117] "RemoveContainer" containerID="7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"
Jul 28 12:38:09.317922 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.317904    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9\": container with ID starting with 7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9 not found: ID does not exist" containerID="7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"
Jul 28 12:38:09.317997 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317928    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"} err="failed to get container status \"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9\": rpc error: code = NotFound desc = could not find container \"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9\": container with ID starting with 7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9 not found: ID does not exist"
Jul 28 12:38:09.317997 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.317948    2563 scope.go:117] "RemoveContainer" containerID="20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"
Jul 28 12:38:09.318198 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.318181    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4\": container with ID starting with 20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4 not found: ID does not exist" containerID="20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"
Jul 28 12:38:09.318240 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318202    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"} err="failed to get container status \"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4\": rpc error: code = NotFound desc = could not find container \"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4\": container with ID starting with 20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4 not found: ID does not exist"
Jul 28 12:38:09.318240 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318215    2563 scope.go:117] "RemoveContainer" containerID="41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"
Jul 28 12:38:09.318438 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:09.318423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d\": container with ID starting with 41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d not found: ID does not exist" containerID="41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"
Jul 28 12:38:09.318503 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318444    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"} err="failed to get container status \"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d\": rpc error: code = NotFound desc = could not find container \"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d\": container with ID starting with 41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d not found: ID does not exist"
Jul 28 12:38:09.318503 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318465    2563 scope.go:117] "RemoveContainer" containerID="5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"
Jul 28 12:38:09.318709 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318694    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089"} err="failed to get container status \"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089\": rpc error: code = NotFound desc = could not find container \"5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089\": container with ID starting with 5c4d74a51495aa174049e943050b542faa9ca7dbb9bc63b18bf4ba820c00d089 not found: ID does not exist"
Jul 28 12:38:09.318768 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318710    2563 scope.go:117] "RemoveContainer" containerID="2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"
Jul 28 12:38:09.318979 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318945    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa"} err="failed to get container status \"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa\": rpc error: code = NotFound desc = could not find container \"2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa\": container with ID starting with 2a29872485ce625f1f3c224c95cddef024e43f372d694a3b41a12457e4c732fa not found: ID does not exist"
Jul 28 12:38:09.318979 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.318970    2563 scope.go:117] "RemoveContainer" containerID="59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"
Jul 28 12:38:09.319284 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.319262    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff"} err="failed to get container status \"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff\": rpc error: code = NotFound desc = could not find container \"59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff\": container with ID starting with 59f1d11682b75c681fd45d35afc2cfd802a5e83e70779d39b526bbdd3a7883ff not found: ID does not exist"
Jul 28 12:38:09.319284 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.319286    2563 scope.go:117] "RemoveContainer" containerID="ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"
Jul 28 12:38:09.319601 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.319581    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9"} err="failed to get container status \"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9\": rpc error: code = NotFound desc = could not find container \"ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9\": container with ID starting with ba58fce82b42f8d06b6c628b8c81be6a20b7558c065c85c6f53bb411f28e19e9 not found: ID does not exist"
Jul 28 12:38:09.319601 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.319600    2563 scope.go:117] "RemoveContainer" containerID="7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"
Jul 28 12:38:09.319931 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.319892    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9"} err="failed to get container status \"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9\": rpc error: code = NotFound desc = could not find container \"7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9\": container with ID starting with 7cd6ce56ed0049cd13c1e40ce910480b7ce538f037d2a896fa0fe785aa4a3fb9 not found: ID does not exist"
Jul 28 12:38:09.320017 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.319933    2563 scope.go:117] "RemoveContainer" containerID="20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"
Jul 28 12:38:09.320216 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.320190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4"} err="failed to get container status \"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4\": rpc error: code = NotFound desc = could not find container \"20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4\": container with ID starting with 20db5b9c1e8c6b49d9e6711f96fdab41e5ba7669c5c1e14445ffe12d7b7d3fc4 not found: ID does not exist"
Jul 28 12:38:09.320305 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.320217    2563 scope.go:117] "RemoveContainer" containerID="41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"
Jul 28 12:38:09.320487 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.320459    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d"} err="failed to get container status \"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d\": rpc error: code = NotFound desc = could not find container \"41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d\": container with ID starting with 41a8b2f38a543d07b2c3bfe24ba120b63f15c80de370e8e518d0fce43655630d not found: ID does not exist"
Jul 28 12:38:09.321058 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321042    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:38:09.321293 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321281    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217ed5d2-02b2-4ca3-95a0-50d207754e72" containerName="registry"
Jul 28 12:38:09.321332 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="217ed5d2-02b2-4ca3-95a0-50d207754e72" containerName="registry"
Jul 28 12:38:09.321332 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321304    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="prom-label-proxy"
Jul 28 12:38:09.321332 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321309    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="prom-label-proxy"
Jul 28 12:38:09.321332 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321319    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-web"
Jul 28 12:38:09.321332 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321325    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-web"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321333    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="alertmanager"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321339    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="alertmanager"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321348    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="config-reloader"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321354    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="config-reloader"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321359    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-metric"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321364    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-metric"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321371    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="init-config-reloader"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321377    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="init-config-reloader"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321383    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" containerName="console"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" containerName="console"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321394    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321398    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321439    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="prom-label-proxy"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321444    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="alertmanager"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321451    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="217ed5d2-02b2-4ca3-95a0-50d207754e72" containerName="registry"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321456    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-metric"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321462    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a281b3d-fa7c-43d5-9975-7f1cc0cbac0e" containerName="console"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321467    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321471    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="config-reloader"
Jul 28 12:38:09.321469 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.321477    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" containerName="kube-rbac-proxy-web"
Jul 28 12:38:09.324687 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.324670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.328072 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328054    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:38:09.328176 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:38:09.328176 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328054    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:38:09.328176 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328161    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:38:09.328347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328256    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:38:09.328347 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4gt2q\""
Jul 28 12:38:09.328498 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:38:09.328559 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328495    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:38:09.328559 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.328500    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:38:09.335255 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.335237    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:38:09.340805 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.340766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:38:09.411206 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411206 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411206 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qk7h\" (UniqueName: \"kubernetes.io/projected/b9cdd796-2a87-4572-9b01-c876f0bd161f-kube-api-access-9qk7h\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411418 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411418 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411418 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-web-config\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411418 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411545 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9cdd796-2a87-4572-9b01-c876f0bd161f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411545 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9cdd796-2a87-4572-9b01-c876f0bd161f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411545 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cdd796-2a87-4572-9b01-c876f0bd161f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411545 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411545 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9cdd796-2a87-4572-9b01-c876f0bd161f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.411689 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.411564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9cdd796-2a87-4572-9b01-c876f0bd161f-config-out\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512453 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9cdd796-2a87-4572-9b01-c876f0bd161f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512556 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9cdd796-2a87-4572-9b01-c876f0bd161f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512556 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cdd796-2a87-4572-9b01-c876f0bd161f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512556 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512556 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9cdd796-2a87-4572-9b01-c876f0bd161f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512812 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9cdd796-2a87-4572-9b01-c876f0bd161f-config-out\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512812 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512812 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.512812 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qk7h\" (UniqueName: \"kubernetes.io/projected/b9cdd796-2a87-4572-9b01-c876f0bd161f-kube-api-access-9qk7h\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513057 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513057 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513057 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-web-config\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513057 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.512994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513261 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.513157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b9cdd796-2a87-4572-9b01-c876f0bd161f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513783 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.513712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9cdd796-2a87-4572-9b01-c876f0bd161f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.513958 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.513938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9cdd796-2a87-4572-9b01-c876f0bd161f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516085 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.515744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9cdd796-2a87-4572-9b01-c876f0bd161f-config-out\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516085 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.515750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9cdd796-2a87-4572-9b01-c876f0bd161f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516085 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.515968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516297 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.516107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516297 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.516171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516404 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.516367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516447 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.516410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.516535 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.516517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.517810 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.517793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9cdd796-2a87-4572-9b01-c876f0bd161f-web-config\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.524183 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.524165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qk7h\" (UniqueName: \"kubernetes.io/projected/b9cdd796-2a87-4572-9b01-c876f0bd161f-kube-api-access-9qk7h\") pod \"alertmanager-main-0\" (UID: \"b9cdd796-2a87-4572-9b01-c876f0bd161f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.636406 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.636360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:38:09.778796 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:09.778762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:38:09.782798 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:38:09.782772    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cdd796_2a87_4572_9b01_c876f0bd161f.slice/crio-58bd0e189ebc478b3310d0244e690865c823204436f7cf5cdf424fc31d574985 WatchSource:0}: Error finding container 58bd0e189ebc478b3310d0244e690865c823204436f7cf5cdf424fc31d574985: Status 404 returned error can't find the container with id 58bd0e189ebc478b3310d0244e690865c823204436f7cf5cdf424fc31d574985
Jul 28 12:38:10.272054 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:10.272022    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9cdd796-2a87-4572-9b01-c876f0bd161f" containerID="158d7948321f10cf4ee8e7ecbf459c2297f9e92577cf11f6ad200e28470b5f6a" exitCode=0
Jul 28 12:38:10.272552 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:10.272057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerDied","Data":"158d7948321f10cf4ee8e7ecbf459c2297f9e92577cf11f6ad200e28470b5f6a"}
Jul 28 12:38:10.272552 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:10.272126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"58bd0e189ebc478b3310d0244e690865c823204436f7cf5cdf424fc31d574985"}
Jul 28 12:38:10.346623 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:10.346597    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3db186b-39a7-4c34-b0e6-c68b59ed6621" path="/var/lib/kubelet/pods/c3db186b-39a7-4c34-b0e6-c68b59ed6621/volumes"
Jul 28 12:38:11.278505 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.278471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"bb2b9abc8385bf728bfb8cc2396dd5b1ce408569ff85d6a84dbe160d49e57169"}
Jul 28 12:38:11.278908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.278541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"001e44f8ab5ebca654875358ef53dc17c0c2f58fb98d9b96ab86b51d982bb8fa"}
Jul 28 12:38:11.278908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.278568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"008a91cf0054f862bbc795b6856b6c16765ced5e48248587cec0655641d065f6"}
Jul 28 12:38:11.278908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.278593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"353fd6b05d5b46722adc5ef4aec473e52a1baa722e2ffd988fb048c302c70de4"}
Jul 28 12:38:11.278908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.278617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"64f7189f4947df1205fbb7757f62b8edc072fac91faa0b24c2afacddd12cec57"}
Jul 28 12:38:11.278908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.278639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b9cdd796-2a87-4572-9b01-c876f0bd161f","Type":"ContainerStarted","Data":"84d8f7facd55fba963a69a880eda1111b2c7635217eac113ab9facdd14436849"}
Jul 28 12:38:11.311334 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:11.311264    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.294222686 podStartE2EDuration="2.311243092s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.020406ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:38:11.308392941 +0000 UTC m=+199.526745653" watchObservedRunningTime="2026-07-28 12:38:11.311243092 +0000 UTC m=+199.529595814"
Jul 28 12:38:13.889561 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:13.889526    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79c58-mzx4c"]
Jul 28 12:38:13.891696 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:13.891680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:13.903485 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:13.903465    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c58-mzx4c"]
Jul 28 12:38:14.051222 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-console-config\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.051389 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glvq\" (UniqueName: \"kubernetes.io/projected/8255ca95-209a-4705-ad83-c4676a928e10-kube-api-access-6glvq\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.051389 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-oauth-config\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.051389 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-serving-cert\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.051389 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-service-ca\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.051515 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-trusted-ca-bundle\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.051515 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.051414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-oauth-serving-cert\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152658 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glvq\" (UniqueName: \"kubernetes.io/projected/8255ca95-209a-4705-ad83-c4676a928e10-kube-api-access-6glvq\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152658 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-oauth-config\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152658 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-serving-cert\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-service-ca\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-trusted-ca-bundle\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-oauth-serving-cert\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.152942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.152770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-console-config\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.153512 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.153489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-oauth-serving-cert\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.153660 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.153638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-service-ca\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.153703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.153687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-trusted-ca-bundle\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.153734 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.153696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-console-config\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.155177 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.155155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-oauth-config\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.155341 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.155324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-serving-cert\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.161566 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.161543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glvq\" (UniqueName: \"kubernetes.io/projected/8255ca95-209a-4705-ad83-c4676a928e10-kube-api-access-6glvq\") pod \"console-79c58-mzx4c\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") " pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.200752 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.200726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:14.329905 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:14.329876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c58-mzx4c"]
Jul 28 12:38:14.333395 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:38:14.333369    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8255ca95_209a_4705_ad83_c4676a928e10.slice/crio-e80d55585052e48d031d8a22ceed9566a041edc36470fffed84266a25196ee67 WatchSource:0}: Error finding container e80d55585052e48d031d8a22ceed9566a041edc36470fffed84266a25196ee67: Status 404 returned error can't find the container with id e80d55585052e48d031d8a22ceed9566a041edc36470fffed84266a25196ee67
Jul 28 12:38:15.297306 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:15.297267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c58-mzx4c" event={"ID":"8255ca95-209a-4705-ad83-c4676a928e10","Type":"ContainerStarted","Data":"d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9"}
Jul 28 12:38:15.297749 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:15.297349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c58-mzx4c" event={"ID":"8255ca95-209a-4705-ad83-c4676a928e10","Type":"ContainerStarted","Data":"e80d55585052e48d031d8a22ceed9566a041edc36470fffed84266a25196ee67"}
Jul 28 12:38:15.316898 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:15.316812    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79c58-mzx4c" podStartSLOduration=2.316795006 podStartE2EDuration="2.316795006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:38:15.315845902 +0000 UTC m=+203.534198614" watchObservedRunningTime="2026-07-28 12:38:15.316795006 +0000 UTC m=+203.535147718"
Jul 28 12:38:24.200894 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:24.200855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:24.201277 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:24.200977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:24.205476 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:24.205455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:24.333284 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:24.333258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:38:24.387366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:24.387328    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6779bb5-lm4rr"]
Jul 28 12:38:49.407543 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.407480    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f6779bb5-lm4rr" podUID="47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" containerName="console" containerID="cri-o://a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39" gracePeriod=15
Jul 28 12:38:49.661765 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.661711    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:38:49.737928 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.737896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-serving-cert\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.737952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-trusted-ca-bundle\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.737979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8b5l\" (UniqueName: \"kubernetes.io/projected/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-kube-api-access-p8b5l\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.737995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-config\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-service-ca\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-oauth-config\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738170 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-oauth-serving-cert\") pod \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\" (UID: \"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd\") "
Jul 28 12:38:49.738594 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:49.738594 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-config" (OuterVolumeSpecName: "console-config") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:49.738594 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:49.738594 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.738542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:49.740331 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.740301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:49.740427 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.740328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:49.740427 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.740376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-kube-api-access-p8b5l" (OuterVolumeSpecName: "kube-api-access-p8b5l") pod "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" (UID: "47cdf838-5722-443e-a5c9-fbc6bd2ca6dd"). InnerVolumeSpecName "kube-api-access-p8b5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:38:49.838599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838570    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:49.838599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838593    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8b5l\" (UniqueName: \"kubernetes.io/projected/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-kube-api-access-p8b5l\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:49.838599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838603    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:49.838599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838611    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-service-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:49.838874 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838619    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-oauth-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:49.838874 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838627    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-oauth-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:49.838874 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:49.838635    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd-console-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:38:50.421585 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.421549    2563 generic.go:356] "Generic (PLEG): container finished" podID="47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" containerID="a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39" exitCode=0
Jul 28 12:38:50.422075 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.421591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6779bb5-lm4rr" event={"ID":"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd","Type":"ContainerDied","Data":"a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39"}
Jul 28 12:38:50.422075 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.421635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6779bb5-lm4rr" event={"ID":"47cdf838-5722-443e-a5c9-fbc6bd2ca6dd","Type":"ContainerDied","Data":"286936e82f60e71f1e239c1d81b66fb9814f70f308c2c7664b97f435a677f6c1"}
Jul 28 12:38:50.422075 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.421662    2563 scope.go:117] "RemoveContainer" containerID="a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39"
Jul 28 12:38:50.422075 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.421594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6779bb5-lm4rr"
Jul 28 12:38:50.430183 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.430157    2563 scope.go:117] "RemoveContainer" containerID="a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39"
Jul 28 12:38:50.430470 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:38:50.430451    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39\": container with ID starting with a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39 not found: ID does not exist" containerID="a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39"
Jul 28 12:38:50.430544 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.430480    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39"} err="failed to get container status \"a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39\": rpc error: code = NotFound desc = could not find container \"a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39\": container with ID starting with a95affef9afb5de886eed46446d98109d6519ff6f1978a85d6e64386e0796c39 not found: ID does not exist"
Jul 28 12:38:50.438215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.438187    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6779bb5-lm4rr"]
Jul 28 12:38:50.442173 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:50.442150    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f6779bb5-lm4rr"]
Jul 28 12:38:52.343809 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:38:52.343766    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" path="/var/lib/kubelet/pods/47cdf838-5722-443e-a5c9-fbc6bd2ca6dd/volumes"
Jul 28 12:39:52.212517 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:39:52.212482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:39:52.216265 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:39:52.216246    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:39:52.221712 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:39:52.221695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:41:08.245608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.245570    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"]
Jul 28 12:41:08.246175 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.245894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" containerName="console"
Jul 28 12:41:08.246175 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.245905    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" containerName="console"
Jul 28 12:41:08.246175 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.245947    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="47cdf838-5722-443e-a5c9-fbc6bd2ca6dd" containerName="console"
Jul 28 12:41:08.247725 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.247710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.250559 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.250533    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 12:41:08.250778 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.250762    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hj574\""
Jul 28 12:41:08.251647 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.251631    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 12:41:08.270723 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.270694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"]
Jul 28 12:41:08.327539 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.327506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.327718 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.327568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.327718 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.327590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbbv\" (UniqueName: \"kubernetes.io/projected/a01671d5-a7d8-460c-9c4d-ba93513dec8e-kube-api-access-kjbbv\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.428387 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.428353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.428560 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.428417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.428560 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.428437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbbv\" (UniqueName: \"kubernetes.io/projected/a01671d5-a7d8-460c-9c4d-ba93513dec8e-kube-api-access-kjbbv\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.428788 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.428767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.428853 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.428788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.437467 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.437438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbbv\" (UniqueName: \"kubernetes.io/projected/a01671d5-a7d8-460c-9c4d-ba93513dec8e-kube-api-access-kjbbv\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.557475 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.557392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:08.697269 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.697238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"]
Jul 28 12:41:08.700839 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:41:08.700792    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01671d5_a7d8_460c_9c4d_ba93513dec8e.slice/crio-02c068ef101e4493fe65ecf8df32edae4ad7431502a4d4b5640439bae5eb6cff WatchSource:0}: Error finding container 02c068ef101e4493fe65ecf8df32edae4ad7431502a4d4b5640439bae5eb6cff: Status 404 returned error can't find the container with id 02c068ef101e4493fe65ecf8df32edae4ad7431502a4d4b5640439bae5eb6cff
Jul 28 12:41:08.703217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.703200    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:41:08.923865 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:08.923757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z" event={"ID":"a01671d5-a7d8-460c-9c4d-ba93513dec8e","Type":"ContainerStarted","Data":"02c068ef101e4493fe65ecf8df32edae4ad7431502a4d4b5640439bae5eb6cff"}
Jul 28 12:41:14.952566 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:14.952527    2563 generic.go:356] "Generic (PLEG): container finished" podID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerID="44ac12b9039055ae3b1862eac43216241d7c7bd1c15251a8d7123ac9b20397c5" exitCode=0
Jul 28 12:41:14.953034 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:14.952618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z" event={"ID":"a01671d5-a7d8-460c-9c4d-ba93513dec8e","Type":"ContainerDied","Data":"44ac12b9039055ae3b1862eac43216241d7c7bd1c15251a8d7123ac9b20397c5"}
Jul 28 12:41:17.967344 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:17.967309    2563 generic.go:356] "Generic (PLEG): container finished" podID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerID="964cb2e377d19527d2e5a7626dc1afe8aad1bf8ad0a3bb121ed36b492c5acb2b" exitCode=0
Jul 28 12:41:17.967745 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:17.967395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z" event={"ID":"a01671d5-a7d8-460c-9c4d-ba93513dec8e","Type":"ContainerDied","Data":"964cb2e377d19527d2e5a7626dc1afe8aad1bf8ad0a3bb121ed36b492c5acb2b"}
Jul 28 12:41:26.000243 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:26.000208    2563 generic.go:356] "Generic (PLEG): container finished" podID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerID="3102810d3adcd45372b631db0c8081167fcb70ec64854f434fd99818d2fbd5dd" exitCode=0
Jul 28 12:41:26.000704 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:26.000275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z" event={"ID":"a01671d5-a7d8-460c-9c4d-ba93513dec8e","Type":"ContainerDied","Data":"3102810d3adcd45372b631db0c8081167fcb70ec64854f434fd99818d2fbd5dd"}
Jul 28 12:41:27.130161 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.130137    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:27.296883 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.296780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjbbv\" (UniqueName: \"kubernetes.io/projected/a01671d5-a7d8-460c-9c4d-ba93513dec8e-kube-api-access-kjbbv\") pod \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") "
Jul 28 12:41:27.297048 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.296891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-util\") pod \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") "
Jul 28 12:41:27.297099 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.297039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-bundle\") pod \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\" (UID: \"a01671d5-a7d8-460c-9c4d-ba93513dec8e\") "
Jul 28 12:41:27.297637 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.297609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-bundle" (OuterVolumeSpecName: "bundle") pod "a01671d5-a7d8-460c-9c4d-ba93513dec8e" (UID: "a01671d5-a7d8-460c-9c4d-ba93513dec8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:41:27.299235 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.299203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01671d5-a7d8-460c-9c4d-ba93513dec8e-kube-api-access-kjbbv" (OuterVolumeSpecName: "kube-api-access-kjbbv") pod "a01671d5-a7d8-460c-9c4d-ba93513dec8e" (UID: "a01671d5-a7d8-460c-9c4d-ba93513dec8e"). InnerVolumeSpecName "kube-api-access-kjbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:41:27.301066 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.301047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-util" (OuterVolumeSpecName: "util") pod "a01671d5-a7d8-460c-9c4d-ba93513dec8e" (UID: "a01671d5-a7d8-460c-9c4d-ba93513dec8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:41:27.398314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.398278    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbbv\" (UniqueName: \"kubernetes.io/projected/a01671d5-a7d8-460c-9c4d-ba93513dec8e-kube-api-access-kjbbv\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:41:27.398314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.398311    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-util\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:41:27.398314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:27.398323    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a01671d5-a7d8-460c-9c4d-ba93513dec8e-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:41:28.009352 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:28.009322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z"
Jul 28 12:41:28.009352 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:28.009322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52t46z" event={"ID":"a01671d5-a7d8-460c-9c4d-ba93513dec8e","Type":"ContainerDied","Data":"02c068ef101e4493fe65ecf8df32edae4ad7431502a4d4b5640439bae5eb6cff"}
Jul 28 12:41:28.009575 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:28.009382    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c068ef101e4493fe65ecf8df32edae4ad7431502a4d4b5640439bae5eb6cff"
Jul 28 12:41:30.948050 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948015    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"]
Jul 28 12:41:30.948588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="extract"
Jul 28 12:41:30.948588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948479    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="extract"
Jul 28 12:41:30.948588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948496    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="util"
Jul 28 12:41:30.948588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948504    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="util"
Jul 28 12:41:30.948588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="pull"
Jul 28 12:41:30.948588 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948520    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="pull"
Jul 28 12:41:30.948889 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.948607    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01671d5-a7d8-460c-9c4d-ba93513dec8e" containerName="extract"
Jul 28 12:41:30.950648 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.950625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:30.953278 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.953258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 12:41:30.953395 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.953374    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 12:41:30.953447 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.953434    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 12:41:30.953528 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.953511    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-xccrw\""
Jul 28 12:41:30.965100 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:30.965078    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"]
Jul 28 12:41:31.131464 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.131428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e4826eb2-96dd-44cc-814e-885d47bf6bca-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz\" (UID: \"e4826eb2-96dd-44cc-814e-885d47bf6bca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.131625 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.131508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27dr\" (UniqueName: \"kubernetes.io/projected/e4826eb2-96dd-44cc-814e-885d47bf6bca-kube-api-access-v27dr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz\" (UID: \"e4826eb2-96dd-44cc-814e-885d47bf6bca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.232563 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.232473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27dr\" (UniqueName: \"kubernetes.io/projected/e4826eb2-96dd-44cc-814e-885d47bf6bca-kube-api-access-v27dr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz\" (UID: \"e4826eb2-96dd-44cc-814e-885d47bf6bca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.232563 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.232523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e4826eb2-96dd-44cc-814e-885d47bf6bca-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz\" (UID: \"e4826eb2-96dd-44cc-814e-885d47bf6bca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.235090 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.235066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e4826eb2-96dd-44cc-814e-885d47bf6bca-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz\" (UID: \"e4826eb2-96dd-44cc-814e-885d47bf6bca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.241893 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.241869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27dr\" (UniqueName: \"kubernetes.io/projected/e4826eb2-96dd-44cc-814e-885d47bf6bca-kube-api-access-v27dr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz\" (UID: \"e4826eb2-96dd-44cc-814e-885d47bf6bca\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.260730 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.260705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:31.396978 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:31.396953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"]
Jul 28 12:41:31.399491 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:41:31.399461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4826eb2_96dd_44cc_814e_885d47bf6bca.slice/crio-f557fced954baa04d165c43064208be35a7966aa19aacc9590dc03603d01dd08 WatchSource:0}: Error finding container f557fced954baa04d165c43064208be35a7966aa19aacc9590dc03603d01dd08: Status 404 returned error can't find the container with id f557fced954baa04d165c43064208be35a7966aa19aacc9590dc03603d01dd08
Jul 28 12:41:32.024990 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:32.024950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz" event={"ID":"e4826eb2-96dd-44cc-814e-885d47bf6bca","Type":"ContainerStarted","Data":"f557fced954baa04d165c43064208be35a7966aa19aacc9590dc03603d01dd08"}
Jul 28 12:41:36.043790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.043757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz" event={"ID":"e4826eb2-96dd-44cc-814e-885d47bf6bca","Type":"ContainerStarted","Data":"c6e0a4f5f13135b3ae4bf9bd4fba3a6f3777dcb164887c3610a850c15b248fe7"}
Jul 28 12:41:36.044175 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.043865    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:41:36.076979 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.076932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz" podStartSLOduration=1.613911388 podStartE2EDuration="6.076916514s" totalImagesPullingTime="4.463005126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:36.073906945 +0000 UTC m=+404.292259657" watchObservedRunningTime="2026-07-28 12:41:36.076916514 +0000 UTC m=+404.295269226"
Jul 28 12:41:36.685042 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.685005    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-tdhgj"]
Jul 28 12:41:36.686994 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.686978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.690139 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.690120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 12:41:36.690268 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.690252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9v58m\""
Jul 28 12:41:36.690646 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.690633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 12:41:36.699195 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.699175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-tdhgj"]
Jul 28 12:41:36.776122 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.776089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fa22f7da-da2d-4ca9-83c2-d78231101156-cabundle0\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.776122 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.776123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.776308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.776150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jct25\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-kube-api-access-jct25\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.877008 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.876965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fa22f7da-da2d-4ca9-83c2-d78231101156-cabundle0\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.877187 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.877016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.877187 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.877055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jct25\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-kube-api-access-jct25\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.877316 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:36.877216    2563 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 12:41:36.877316 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:36.877241    2563 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:41:36.877316 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:36.877253    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:41:36.877316 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:36.877270    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-tdhgj: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:41:36.877546 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:36.877347    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates podName:fa22f7da-da2d-4ca9-83c2-d78231101156 nodeName:}" failed. No retries permitted until 2026-07-28 12:41:37.377327103 +0000 UTC m=+405.595679798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates") pod "keda-operator-77f648d694-tdhgj" (UID: "fa22f7da-da2d-4ca9-83c2-d78231101156") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 12:41:36.877737 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.877715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fa22f7da-da2d-4ca9-83c2-d78231101156-cabundle0\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:36.888474 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:36.888449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jct25\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-kube-api-access-jct25\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:37.212929 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.212884    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"]
Jul 28 12:41:37.215086 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.215069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.217636 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.217612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 12:41:37.225893 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.225865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"]
Jul 28 12:41:37.279989 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.279957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/14e6bb16-be99-4295-a1a0-23404d84551e-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.280168 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.280012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-certificates\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.280168 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.280041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdl6z\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-kube-api-access-hdl6z\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.380732 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.380704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-certificates\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.380732 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.380746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdl6z\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-kube-api-access-hdl6z\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.380792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.380877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/14e6bb16-be99-4295-a1a0-23404d84551e-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.380883    2563 secret.go:281] references non-existent secret key: tls.crt
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.380925    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.380954    2563 secret.go:281] references non-existent secret key: ca.crt
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.380963    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl: references non-existent secret key: tls.crt
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.380976    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 12:41:37.381011 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.380988    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-tdhgj: references non-existent secret key: ca.crt
Jul 28 12:41:37.381371 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.381035    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-certificates podName:14e6bb16-be99-4295-a1a0-23404d84551e nodeName:}" failed. No retries permitted until 2026-07-28 12:41:37.881015924 +0000 UTC m=+406.099368628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-certificates") pod "keda-metrics-apiserver-55557df8c-5bsnl" (UID: "14e6bb16-be99-4295-a1a0-23404d84551e") : references non-existent secret key: tls.crt
Jul 28 12:41:37.381371 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:41:37.381060    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates podName:fa22f7da-da2d-4ca9-83c2-d78231101156 nodeName:}" failed. No retries permitted until 2026-07-28 12:41:38.381041285 +0000 UTC m=+406.599394000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates") pod "keda-operator-77f648d694-tdhgj" (UID: "fa22f7da-da2d-4ca9-83c2-d78231101156") : references non-existent secret key: ca.crt
Jul 28 12:41:37.381371 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.381291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/14e6bb16-be99-4295-a1a0-23404d84551e-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.392269 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.392239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdl6z\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-kube-api-access-hdl6z\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.671560 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.671482    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-5nhxq"]
Jul 28 12:41:37.673611 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.673591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.675912 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.675886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 12:41:37.682998 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.682968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7734b9da-8b18-486b-a973-e30a54fa43f9-certificates\") pod \"keda-admission-56f9798bb4-5nhxq\" (UID: \"7734b9da-8b18-486b-a973-e30a54fa43f9\") " pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.683176 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.683158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8lz\" (UniqueName: \"kubernetes.io/projected/7734b9da-8b18-486b-a973-e30a54fa43f9-kube-api-access-ck8lz\") pod \"keda-admission-56f9798bb4-5nhxq\" (UID: \"7734b9da-8b18-486b-a973-e30a54fa43f9\") " pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.684251 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.684224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-5nhxq"]
Jul 28 12:41:37.783711 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.783675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7734b9da-8b18-486b-a973-e30a54fa43f9-certificates\") pod \"keda-admission-56f9798bb4-5nhxq\" (UID: \"7734b9da-8b18-486b-a973-e30a54fa43f9\") " pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.783711 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.783712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8lz\" (UniqueName: \"kubernetes.io/projected/7734b9da-8b18-486b-a973-e30a54fa43f9-kube-api-access-ck8lz\") pod \"keda-admission-56f9798bb4-5nhxq\" (UID: \"7734b9da-8b18-486b-a973-e30a54fa43f9\") " pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.786362 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.786336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7734b9da-8b18-486b-a973-e30a54fa43f9-certificates\") pod \"keda-admission-56f9798bb4-5nhxq\" (UID: \"7734b9da-8b18-486b-a973-e30a54fa43f9\") " pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.793038 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.793016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8lz\" (UniqueName: \"kubernetes.io/projected/7734b9da-8b18-486b-a973-e30a54fa43f9-kube-api-access-ck8lz\") pod \"keda-admission-56f9798bb4-5nhxq\" (UID: \"7734b9da-8b18-486b-a973-e30a54fa43f9\") " pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:37.884496 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.884460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-certificates\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.887418 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.887391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/14e6bb16-be99-4295-a1a0-23404d84551e-certificates\") pod \"keda-metrics-apiserver-55557df8c-5bsnl\" (UID: \"14e6bb16-be99-4295-a1a0-23404d84551e\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:37.985529 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:37.985491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:38.126085 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.125722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:38.139524 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.139448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-5nhxq"]
Jul 28 12:41:38.143990 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:41:38.143931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7734b9da_8b18_486b_a973_e30a54fa43f9.slice/crio-86e971d198830e26a1f4f64a965d8b3d68cf2a34fd27b8be257530634cf154b4 WatchSource:0}: Error finding container 86e971d198830e26a1f4f64a965d8b3d68cf2a34fd27b8be257530634cf154b4: Status 404 returned error can't find the container with id 86e971d198830e26a1f4f64a965d8b3d68cf2a34fd27b8be257530634cf154b4
Jul 28 12:41:38.271727 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.271642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"]
Jul 28 12:41:38.274752 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:41:38.274725    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e6bb16_be99_4295_a1a0_23404d84551e.slice/crio-20e3a96fb0d379bdada0771d0d0cb818c2353854b8a171d4b7e3cd3fd96ad4d6 WatchSource:0}: Error finding container 20e3a96fb0d379bdada0771d0d0cb818c2353854b8a171d4b7e3cd3fd96ad4d6: Status 404 returned error can't find the container with id 20e3a96fb0d379bdada0771d0d0cb818c2353854b8a171d4b7e3cd3fd96ad4d6
Jul 28 12:41:38.388730 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.388691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:38.391262 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.391242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fa22f7da-da2d-4ca9-83c2-d78231101156-certificates\") pod \"keda-operator-77f648d694-tdhgj\" (UID: \"fa22f7da-da2d-4ca9-83c2-d78231101156\") " pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:38.496976 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.496938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:38.626069 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:38.626040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-tdhgj"]
Jul 28 12:41:38.628926 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:41:38.628901    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa22f7da_da2d_4ca9_83c2_d78231101156.slice/crio-84253a39c167d6ae2f768b9c048e1882d8ca919a3a27ea35801919bdd5356145 WatchSource:0}: Error finding container 84253a39c167d6ae2f768b9c048e1882d8ca919a3a27ea35801919bdd5356145: Status 404 returned error can't find the container with id 84253a39c167d6ae2f768b9c048e1882d8ca919a3a27ea35801919bdd5356145
Jul 28 12:41:39.058877 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:39.058841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl" event={"ID":"14e6bb16-be99-4295-a1a0-23404d84551e","Type":"ContainerStarted","Data":"20e3a96fb0d379bdada0771d0d0cb818c2353854b8a171d4b7e3cd3fd96ad4d6"}
Jul 28 12:41:39.060087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:39.060062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-tdhgj" event={"ID":"fa22f7da-da2d-4ca9-83c2-d78231101156","Type":"ContainerStarted","Data":"84253a39c167d6ae2f768b9c048e1882d8ca919a3a27ea35801919bdd5356145"}
Jul 28 12:41:39.061142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:39.061124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq" event={"ID":"7734b9da-8b18-486b-a973-e30a54fa43f9","Type":"ContainerStarted","Data":"86e971d198830e26a1f4f64a965d8b3d68cf2a34fd27b8be257530634cf154b4"}
Jul 28 12:41:41.071942 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:41.071854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq" event={"ID":"7734b9da-8b18-486b-a973-e30a54fa43f9","Type":"ContainerStarted","Data":"0ef071fa36d6db5b3119b4d320379696d15f341004f4c1c3c245603dbf3babfe"}
Jul 28 12:41:41.072634 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:41.071949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:41:41.093512 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:41.093453    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq" podStartSLOduration=2.048604446 podStartE2EDuration="4.093427055s" totalImagesPullingTime="2.044822609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:41.090094352 +0000 UTC m=+409.308447065" watchObservedRunningTime="2026-07-28 12:41:41.093427055 +0000 UTC m=+409.311779768"
Jul 28 12:41:44.088181 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:44.088139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-tdhgj" event={"ID":"fa22f7da-da2d-4ca9-83c2-d78231101156","Type":"ContainerStarted","Data":"6996f2b040810c2ef7e9a549055fdebc582c7b43cd8d90b5b56aee103da0fa41"}
Jul 28 12:41:44.088629 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:44.088208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:41:44.089577 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:44.089557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl" event={"ID":"14e6bb16-be99-4295-a1a0-23404d84551e","Type":"ContainerStarted","Data":"3ce35a71797543a465e7b85c95e93797a66cda2b85daf7b1f2c13b98fdcd737e"}
Jul 28 12:41:44.089688 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:44.089678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:44.111251 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:44.111203    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-tdhgj" podStartSLOduration=3.511124299 podStartE2EDuration="8.111188939s" totalImagesPullingTime="4.60006464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:44.110218996 +0000 UTC m=+412.328571709" watchObservedRunningTime="2026-07-28 12:41:44.111188939 +0000 UTC m=+412.329541651"
Jul 28 12:41:55.099067 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:55.099039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl"
Jul 28 12:41:55.119358 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:55.119304    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-5bsnl" podStartSLOduration=13.168845242 podStartE2EDuration="18.119290494s" totalImagesPullingTime="4.950445252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:41:44.132124598 +0000 UTC m=+412.350477311" watchObservedRunningTime="2026-07-28 12:41:55.119290494 +0000 UTC m=+423.337643207"
Jul 28 12:41:57.050310 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:41:57.050281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-l8wsz"
Jul 28 12:42:02.080328 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:02.080289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-5nhxq"
Jul 28 12:42:05.097074 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:05.097048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-tdhgj"
Jul 28 12:42:11.618661 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:11.618624    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79c58-mzx4c"]
Jul 28 12:42:36.638259 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.638203    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79c58-mzx4c" podUID="8255ca95-209a-4705-ad83-c4676a928e10" containerName="console" containerID="cri-o://d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9" gracePeriod=15
Jul 28 12:42:36.879414 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.879390    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:42:36.977575 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-oauth-serving-cert\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.977770 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977580    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6glvq\" (UniqueName: \"kubernetes.io/projected/8255ca95-209a-4705-ad83-c4676a928e10-kube-api-access-6glvq\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.977770 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-trusted-ca-bundle\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.977770 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-oauth-config\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.977770 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977682    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-service-ca\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.977770 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-serving-cert\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.978070 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.977772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-console-config\") pod \"8255ca95-209a-4705-ad83-c4676a928e10\" (UID: \"8255ca95-209a-4705-ad83-c4676a928e10\") "
Jul 28 12:42:36.978142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.978108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:36.978259 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.978231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:36.978344 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.978282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-service-ca" (OuterVolumeSpecName: "service-ca") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:36.978344 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.978306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-console-config" (OuterVolumeSpecName: "console-config") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:42:36.980056 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.980030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:42:36.980153 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.980048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8255ca95-209a-4705-ad83-c4676a928e10-kube-api-access-6glvq" (OuterVolumeSpecName: "kube-api-access-6glvq") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "kube-api-access-6glvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:42:36.980153 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:36.980063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8255ca95-209a-4705-ad83-c4676a928e10" (UID: "8255ca95-209a-4705-ad83-c4676a928e10"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:42:37.078379 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078346    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-oauth-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.078379 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078374    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6glvq\" (UniqueName: \"kubernetes.io/projected/8255ca95-209a-4705-ad83-c4676a928e10-kube-api-access-6glvq\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.078379 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078383    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-trusted-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.078379 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078392    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-oauth-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.078641 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078400    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-service-ca\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.078641 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078409    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8255ca95-209a-4705-ad83-c4676a928e10-console-serving-cert\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.078641 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.078417    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8255ca95-209a-4705-ad83-c4676a928e10-console-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:42:37.319355 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.319264    2563 generic.go:356] "Generic (PLEG): container finished" podID="8255ca95-209a-4705-ad83-c4676a928e10" containerID="d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9" exitCode=0
Jul 28 12:42:37.319355 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.319304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c58-mzx4c" event={"ID":"8255ca95-209a-4705-ad83-c4676a928e10","Type":"ContainerDied","Data":"d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9"}
Jul 28 12:42:37.319355 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.319316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c58-mzx4c"
Jul 28 12:42:37.319355 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.319340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c58-mzx4c" event={"ID":"8255ca95-209a-4705-ad83-c4676a928e10","Type":"ContainerDied","Data":"e80d55585052e48d031d8a22ceed9566a041edc36470fffed84266a25196ee67"}
Jul 28 12:42:37.319355 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.319358    2563 scope.go:117] "RemoveContainer" containerID="d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9"
Jul 28 12:42:37.328619 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.328603    2563 scope.go:117] "RemoveContainer" containerID="d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9"
Jul 28 12:42:37.328898 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:42:37.328875    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9\": container with ID starting with d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9 not found: ID does not exist" containerID="d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9"
Jul 28 12:42:37.328969 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.328906    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9"} err="failed to get container status \"d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9\": rpc error: code = NotFound desc = could not find container \"d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9\": container with ID starting with d5472e5f5a100f3bdfddeeb42686cc6a67711e44bed88bb1ef9bb0d7c005a1b9 not found: ID does not exist"
Jul 28 12:42:37.346314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.346291    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79c58-mzx4c"]
Jul 28 12:42:37.351674 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:37.351652    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79c58-mzx4c"]
Jul 28 12:42:38.343709 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:38.343666    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8255ca95-209a-4705-ad83-c4676a928e10" path="/var/lib/kubelet/pods/8255ca95-209a-4705-ad83-c4676a928e10/volumes"
Jul 28 12:42:56.614710 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.614674    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-qhbmp"]
Jul 28 12:42:56.615136 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.614986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8255ca95-209a-4705-ad83-c4676a928e10" containerName="console"
Jul 28 12:42:56.615136 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.614997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8255ca95-209a-4705-ad83-c4676a928e10" containerName="console"
Jul 28 12:42:56.615136 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.615080    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8255ca95-209a-4705-ad83-c4676a928e10" containerName="console"
Jul 28 12:42:56.616980 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.616964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.619436 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.619403    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 12:42:56.619701 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.619685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-4qcvp\""
Jul 28 12:42:56.620415 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.620399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 12:42:56.620476 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.620433    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 12:42:56.627426 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.627401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-qhbmp"]
Jul 28 12:42:56.707525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.707485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfgt\" (UniqueName: \"kubernetes.io/projected/3f980bfb-c1f5-46e3-99cc-970589383c71-kube-api-access-2hfgt\") pod \"llmisvc-controller-manager-654c557856-qhbmp\" (UID: \"3f980bfb-c1f5-46e3-99cc-970589383c71\") " pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.707525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.707523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f980bfb-c1f5-46e3-99cc-970589383c71-cert\") pod \"llmisvc-controller-manager-654c557856-qhbmp\" (UID: \"3f980bfb-c1f5-46e3-99cc-970589383c71\") " pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.808750 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.808711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfgt\" (UniqueName: \"kubernetes.io/projected/3f980bfb-c1f5-46e3-99cc-970589383c71-kube-api-access-2hfgt\") pod \"llmisvc-controller-manager-654c557856-qhbmp\" (UID: \"3f980bfb-c1f5-46e3-99cc-970589383c71\") " pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.808750 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.808753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f980bfb-c1f5-46e3-99cc-970589383c71-cert\") pod \"llmisvc-controller-manager-654c557856-qhbmp\" (UID: \"3f980bfb-c1f5-46e3-99cc-970589383c71\") " pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.811430 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.811401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f980bfb-c1f5-46e3-99cc-970589383c71-cert\") pod \"llmisvc-controller-manager-654c557856-qhbmp\" (UID: \"3f980bfb-c1f5-46e3-99cc-970589383c71\") " pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.817934 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.817911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfgt\" (UniqueName: \"kubernetes.io/projected/3f980bfb-c1f5-46e3-99cc-970589383c71-kube-api-access-2hfgt\") pod \"llmisvc-controller-manager-654c557856-qhbmp\" (UID: \"3f980bfb-c1f5-46e3-99cc-970589383c71\") " pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:56.927447 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:56.927361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:57.060434 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:57.060399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-qhbmp"]
Jul 28 12:42:57.065860 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:42:57.065804    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f980bfb_c1f5_46e3_99cc_970589383c71.slice/crio-195845e4a113de56ac37b53431471caecf916de0c9a59c737a4f742e5f726013 WatchSource:0}: Error finding container 195845e4a113de56ac37b53431471caecf916de0c9a59c737a4f742e5f726013: Status 404 returned error can't find the container with id 195845e4a113de56ac37b53431471caecf916de0c9a59c737a4f742e5f726013
Jul 28 12:42:57.400478 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:57.400443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp" event={"ID":"3f980bfb-c1f5-46e3-99cc-970589383c71","Type":"ContainerStarted","Data":"195845e4a113de56ac37b53431471caecf916de0c9a59c737a4f742e5f726013"}
Jul 28 12:42:59.411253 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:59.411221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp" event={"ID":"3f980bfb-c1f5-46e3-99cc-970589383c71","Type":"ContainerStarted","Data":"4ff3bec09d246e4a8c7403f223177a33e77b8775ff51ada4b3cd6d602b7aad70"}
Jul 28 12:42:59.411636 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:59.411310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:42:59.429814 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:42:59.429752    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp" podStartSLOduration=1.223694237 podStartE2EDuration="3.429730593s" totalImagesPullingTime="2.206036356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:59.429252837 +0000 UTC m=+487.647605576" watchObservedRunningTime="2026-07-28 12:42:59.429730593 +0000 UTC m=+487.648083305"
Jul 28 12:43:30.418428 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:43:30.418398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-qhbmp"
Jul 28 12:44:13.003959 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.003923    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2t7m7"]
Jul 28 12:44:13.007062 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.007045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.010590 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.010571    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-k5sqt\""
Jul 28 12:44:13.011110 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.011094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 12:44:13.076722 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.076689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2t7m7"]
Jul 28 12:44:13.119480 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.119443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdsh6\" (UniqueName: \"kubernetes.io/projected/52594f10-5bd2-4687-b357-b0038becdca8-kube-api-access-sdsh6\") pod \"model-serving-api-7b954ffc95-2t7m7\" (UID: \"52594f10-5bd2-4687-b357-b0038becdca8\") " pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.119480 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.119486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52594f10-5bd2-4687-b357-b0038becdca8-tls-certs\") pod \"model-serving-api-7b954ffc95-2t7m7\" (UID: \"52594f10-5bd2-4687-b357-b0038becdca8\") " pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.220498 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.220455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdsh6\" (UniqueName: \"kubernetes.io/projected/52594f10-5bd2-4687-b357-b0038becdca8-kube-api-access-sdsh6\") pod \"model-serving-api-7b954ffc95-2t7m7\" (UID: \"52594f10-5bd2-4687-b357-b0038becdca8\") " pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.220692 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.220514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52594f10-5bd2-4687-b357-b0038becdca8-tls-certs\") pod \"model-serving-api-7b954ffc95-2t7m7\" (UID: \"52594f10-5bd2-4687-b357-b0038becdca8\") " pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.223174 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.223153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52594f10-5bd2-4687-b357-b0038becdca8-tls-certs\") pod \"model-serving-api-7b954ffc95-2t7m7\" (UID: \"52594f10-5bd2-4687-b357-b0038becdca8\") " pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.229982 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.229960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdsh6\" (UniqueName: \"kubernetes.io/projected/52594f10-5bd2-4687-b357-b0038becdca8-kube-api-access-sdsh6\") pod \"model-serving-api-7b954ffc95-2t7m7\" (UID: \"52594f10-5bd2-4687-b357-b0038becdca8\") " pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.317163 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.317068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:13.458729 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.458706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2t7m7"]
Jul 28 12:44:13.461138 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:44:13.461116    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52594f10_5bd2_4687_b357_b0038becdca8.slice/crio-1f9214b3cfee5144005c00fe22a460b7a3edd4fa97c2caaba37a5391b3362912 WatchSource:0}: Error finding container 1f9214b3cfee5144005c00fe22a460b7a3edd4fa97c2caaba37a5391b3362912: Status 404 returned error can't find the container with id 1f9214b3cfee5144005c00fe22a460b7a3edd4fa97c2caaba37a5391b3362912
Jul 28 12:44:13.716530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:13.716494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2t7m7" event={"ID":"52594f10-5bd2-4687-b357-b0038becdca8","Type":"ContainerStarted","Data":"1f9214b3cfee5144005c00fe22a460b7a3edd4fa97c2caaba37a5391b3362912"}
Jul 28 12:44:16.733685 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:16.733650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2t7m7" event={"ID":"52594f10-5bd2-4687-b357-b0038becdca8","Type":"ContainerStarted","Data":"7c7bb8812b16bb9652a8d9199b5e1e252afeadada13029bc311e113d17a83315"}
Jul 28 12:44:16.734102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:16.733783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:16.753517 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:16.753456    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2t7m7" podStartSLOduration=2.086833367 podStartE2EDuration="4.753431601s" totalImagesPullingTime="2.666598234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:16.752226165 +0000 UTC m=+564.970578878" watchObservedRunningTime="2026-07-28 12:44:16.753431601 +0000 UTC m=+564.971784313"
Jul 28 12:44:27.741964 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:27.741938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2t7m7"
Jul 28 12:44:52.245244 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:52.245216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:44:52.247869 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:44:52.247846    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:46:00.856940 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.856898    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"]
Jul 28 12:46:00.860518 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.860498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:00.862873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.862850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9baeb-predictor-serving-cert\""
Jul 28 12:46:00.862873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.862869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9baeb-kube-rbac-proxy-sar-config\""
Jul 28 12:46:00.863843 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.863809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mdrhx\""
Jul 28 12:46:00.863918 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.863812    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:46:00.863918 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.863870    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:46:00.869723 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.869704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"]
Jul 28 12:46:00.900854 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.900811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:00.901001 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.900869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ldd\" (UniqueName: \"kubernetes.io/projected/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-api-access-b6ldd\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:00.901001 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:00.900953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.001580 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.001540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.001744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.001607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.001744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.001632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ldd\" (UniqueName: \"kubernetes.io/projected/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-api-access-b6ldd\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.001873 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:46:01.001767    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-serving-cert: secret "error-404-isvc-9baeb-predictor-serving-cert" not found
Jul 28 12:46:01.001873 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:46:01.001856    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls podName:570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73 nodeName:}" failed. No retries permitted until 2026-07-28 12:46:01.501813582 +0000 UTC m=+669.720166271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls") pod "error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" (UID: "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73") : secret "error-404-isvc-9baeb-predictor-serving-cert" not found
Jul 28 12:46:01.002367 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.002344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.016986 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.016962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ldd\" (UniqueName: \"kubernetes.io/projected/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-api-access-b6ldd\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.506481 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.506444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.509047 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.509029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls\") pod \"error-404-isvc-9baeb-predictor-85d6f87596-8qwgd\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") " pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.771447 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.771367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:01.909325 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:01.909298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"]
Jul 28 12:46:02.166614 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:02.166528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" event={"ID":"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73","Type":"ContainerStarted","Data":"c9eebd8eb12856972aa6b8e3451eec4a1073a87c55cacb62f89497822c84ab09"}
Jul 28 12:46:13.216907 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:13.216886    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:46:13.228833 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:13.228791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" event={"ID":"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73","Type":"ContainerStarted","Data":"f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d"}
Jul 28 12:46:16.244966 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:16.244934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" event={"ID":"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73","Type":"ContainerStarted","Data":"628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b"}
Jul 28 12:46:16.245380 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:16.244994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:16.245380 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:16.245050    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:16.246572 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:16.246541    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:46:16.265683 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:16.265633    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podStartSLOduration=2.854554732 podStartE2EDuration="16.265616816s" totalImagesPullingTime="13.411062084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:46:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:16.264627727 +0000 UTC m=+684.482980440" watchObservedRunningTime="2026-07-28 12:46:16.265616816 +0000 UTC m=+684.483969532"
Jul 28 12:46:17.249609 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:17.249567    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:46:22.254598 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:22.254570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:46:22.255041 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:22.255013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:46:32.255922 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:32.255877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:46:42.255260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:42.255219    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:46:52.255140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:46:52.255056    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:47:02.256614 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:02.256584    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:47:30.558842 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.558789    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"]
Jul 28 12:47:30.560927 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.560912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:30.563586 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.563563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-9baeb-kube-rbac-proxy-sar-config\""
Jul 28 12:47:30.563693 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.563565    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-9baeb-serving-cert\""
Jul 28 12:47:30.571032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.571006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"]
Jul 28 12:47:30.657087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.657055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9578c1e6-0abc-4744-a9e9-d1078048a18b-openshift-service-ca-bundle\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:30.657087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.657087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:30.758087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.758046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9578c1e6-0abc-4744-a9e9-d1078048a18b-openshift-service-ca-bundle\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:30.758087 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.758086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:30.758295 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:47:30.758205    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-9baeb-serving-cert: secret "switch-graph-9baeb-serving-cert" not found
Jul 28 12:47:30.758295 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:47:30.758271    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls podName:9578c1e6-0abc-4744-a9e9-d1078048a18b nodeName:}" failed. No retries permitted until 2026-07-28 12:47:31.258253731 +0000 UTC m=+759.476606420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls") pod "switch-graph-9baeb-7554d76cc8-9rcpx" (UID: "9578c1e6-0abc-4744-a9e9-d1078048a18b") : secret "switch-graph-9baeb-serving-cert" not found
Jul 28 12:47:30.758707 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:30.758674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9578c1e6-0abc-4744-a9e9-d1078048a18b-openshift-service-ca-bundle\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:31.262037 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:31.261988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:31.264518 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:31.264488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls\") pod \"switch-graph-9baeb-7554d76cc8-9rcpx\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") " pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:31.471236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:31.471201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:31.615592 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:31.615561    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"]
Jul 28 12:47:31.618598 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:47:31.618578    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9578c1e6_0abc_4744_a9e9_d1078048a18b.slice/crio-5695caa1107b773a4640b8c5fcaac16dd5d9556f47aa49fb40a1a4568fc77a2a WatchSource:0}: Error finding container 5695caa1107b773a4640b8c5fcaac16dd5d9556f47aa49fb40a1a4568fc77a2a: Status 404 returned error can't find the container with id 5695caa1107b773a4640b8c5fcaac16dd5d9556f47aa49fb40a1a4568fc77a2a
Jul 28 12:47:32.567778 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:32.567739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" event={"ID":"9578c1e6-0abc-4744-a9e9-d1078048a18b","Type":"ContainerStarted","Data":"5695caa1107b773a4640b8c5fcaac16dd5d9556f47aa49fb40a1a4568fc77a2a"}
Jul 28 12:47:34.578342 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:34.578309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" event={"ID":"9578c1e6-0abc-4744-a9e9-d1078048a18b","Type":"ContainerStarted","Data":"e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33"}
Jul 28 12:47:34.578708 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:34.578447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:34.597262 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:34.597216    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podStartSLOduration=2.158296563 podStartE2EDuration="4.597203437s" totalImagesPullingTime="2.438906874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:47:34.594814911 +0000 UTC m=+762.813167635" watchObservedRunningTime="2026-07-28 12:47:34.597203437 +0000 UTC m=+762.815556148"
Jul 28 12:47:40.589488 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:40.589459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:40.859547 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:40.859467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"]
Jul 28 12:47:40.859757 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:40.859706    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" containerID="cri-o://e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33" gracePeriod=30
Jul 28 12:47:41.229354 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.229315    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"]
Jul 28 12:47:41.231702 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.231685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.235028 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.235004    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fc405-predictor-serving-cert\""
Jul 28 12:47:41.235610 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.235578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fc405-kube-rbac-proxy-sar-config\""
Jul 28 12:47:41.245192 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.245171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"]
Jul 28 12:47:41.286522 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.286493    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"]
Jul 28 12:47:41.286738 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.286712    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" containerID="cri-o://f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d" gracePeriod=30
Jul 28 12:47:41.286796 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.286739    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kube-rbac-proxy" containerID="cri-o://628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b" gracePeriod=30
Jul 28 12:47:41.343625 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.343593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.343769 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.343643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45r9s\" (UniqueName: \"kubernetes.io/projected/377a0412-19a2-43c5-a249-a7edecac9337-kube-api-access-45r9s\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.343769 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.343756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/377a0412-19a2-43c5-a249-a7edecac9337-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.445154 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.445111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/377a0412-19a2-43c5-a249-a7edecac9337-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.445346 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.445165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.445346 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.445200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45r9s\" (UniqueName: \"kubernetes.io/projected/377a0412-19a2-43c5-a249-a7edecac9337-kube-api-access-45r9s\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.445346 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:47:41.445328    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fc405-predictor-serving-cert: secret "error-404-isvc-fc405-predictor-serving-cert" not found
Jul 28 12:47:41.445504 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:47:41.445405    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls podName:377a0412-19a2-43c5-a249-a7edecac9337 nodeName:}" failed. No retries permitted until 2026-07-28 12:47:41.945389021 +0000 UTC m=+770.163741712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls") pod "error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" (UID: "377a0412-19a2-43c5-a249-a7edecac9337") : secret "error-404-isvc-fc405-predictor-serving-cert" not found
Jul 28 12:47:41.445806 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.445786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/377a0412-19a2-43c5-a249-a7edecac9337-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.463067 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.463014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45r9s\" (UniqueName: \"kubernetes.io/projected/377a0412-19a2-43c5-a249-a7edecac9337-kube-api-access-45r9s\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.610774 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.610692    2563 generic.go:356] "Generic (PLEG): container finished" podID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerID="628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b" exitCode=2
Jul 28 12:47:41.610774 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.610754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" event={"ID":"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73","Type":"ContainerDied","Data":"628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b"}
Jul 28 12:47:41.950887 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.950854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:41.953496 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:41.953460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls\") pod \"error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") " pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:42.141740 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.141699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:42.250577 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.250541    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 28 12:47:42.256114 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.256077    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 12:47:42.276998 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.276974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"]
Jul 28 12:47:42.279502 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:47:42.279476    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377a0412_19a2_43c5_a249_a7edecac9337.slice/crio-c5c0e7867dff585d5b8acba0bc9dafdb3d9f44adffe0f98c06b039b9705160c3 WatchSource:0}: Error finding container c5c0e7867dff585d5b8acba0bc9dafdb3d9f44adffe0f98c06b039b9705160c3: Status 404 returned error can't find the container with id c5c0e7867dff585d5b8acba0bc9dafdb3d9f44adffe0f98c06b039b9705160c3
Jul 28 12:47:42.616691 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.616601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" event={"ID":"377a0412-19a2-43c5-a249-a7edecac9337","Type":"ContainerStarted","Data":"8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6"}
Jul 28 12:47:42.616691 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.616659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" event={"ID":"377a0412-19a2-43c5-a249-a7edecac9337","Type":"ContainerStarted","Data":"7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4"}
Jul 28 12:47:42.616691 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.616690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:42.617294 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.616719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:42.617294 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.616742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" event={"ID":"377a0412-19a2-43c5-a249-a7edecac9337","Type":"ContainerStarted","Data":"c5c0e7867dff585d5b8acba0bc9dafdb3d9f44adffe0f98c06b039b9705160c3"}
Jul 28 12:47:42.618448 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.618419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:47:42.636574 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:42.636517    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podStartSLOduration=1.636495461 podStartE2EDuration="1.636495461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:47:42.636144487 +0000 UTC m=+770.854497211" watchObservedRunningTime="2026-07-28 12:47:42.636495461 +0000 UTC m=+770.854848174"
Jul 28 12:47:43.621045 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:43.621012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:47:45.587502 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:45.587458    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:47.250691 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.250634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 28 12:47:47.630734 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.630710    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:47:47.640070 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.640037    2563 generic.go:356] "Generic (PLEG): container finished" podID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerID="f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d" exitCode=0
Jul 28 12:47:47.640217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.640082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" event={"ID":"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73","Type":"ContainerDied","Data":"f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d"}
Jul 28 12:47:47.640217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.640100    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"
Jul 28 12:47:47.640217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.640138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd" event={"ID":"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73","Type":"ContainerDied","Data":"c9eebd8eb12856972aa6b8e3451eec4a1073a87c55cacb62f89497822c84ab09"}
Jul 28 12:47:47.640217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.640173    2563 scope.go:117] "RemoveContainer" containerID="628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b"
Jul 28 12:47:47.648462 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.648442    2563 scope.go:117] "RemoveContainer" containerID="f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d"
Jul 28 12:47:47.656011 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.655991    2563 scope.go:117] "RemoveContainer" containerID="628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b"
Jul 28 12:47:47.656295 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:47:47.656275    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b\": container with ID starting with 628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b not found: ID does not exist" containerID="628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b"
Jul 28 12:47:47.656339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.656305    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b"} err="failed to get container status \"628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b\": rpc error: code = NotFound desc = could not find container \"628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b\": container with ID starting with 628c3bffea649a369bfadc6cfcabaf1e1c1a32958397ff94a4e0d736badc7c8b not found: ID does not exist"
Jul 28 12:47:47.656339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.656324    2563 scope.go:117] "RemoveContainer" containerID="f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d"
Jul 28 12:47:47.656579 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:47:47.656557    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d\": container with ID starting with f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d not found: ID does not exist" containerID="f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d"
Jul 28 12:47:47.656623 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.656587    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d"} err="failed to get container status \"f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d\": rpc error: code = NotFound desc = could not find container \"f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d\": container with ID starting with f8ba53e1d73595e52caf8a741fc9a59aadc0bc192e36ad70d41dc2c190d8c68d not found: ID does not exist"
Jul 28 12:47:47.701842 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.701781    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-rbac-proxy-sar-config\") pod \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") "
Jul 28 12:47:47.702025 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.701876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls\") pod \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") "
Jul 28 12:47:47.702025 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.701949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6ldd\" (UniqueName: \"kubernetes.io/projected/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-api-access-b6ldd\") pod \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\" (UID: \"570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73\") "
Jul 28 12:47:47.702252 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.702227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" (UID: "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:47:47.704288 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.704257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" (UID: "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:47:47.704403 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.704312    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-api-access-b6ldd" (OuterVolumeSpecName: "kube-api-access-b6ldd") pod "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" (UID: "570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73"). InnerVolumeSpecName "kube-api-access-b6ldd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:47:47.803224 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.803136    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6ldd\" (UniqueName: \"kubernetes.io/projected/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-api-access-b6ldd\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:47:47.803224 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.803166    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:47:47.803224 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.803175    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:47:47.968490 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.968455    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"]
Jul 28 12:47:47.978022 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:47.977986    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9baeb-predictor-85d6f87596-8qwgd"]
Jul 28 12:47:48.344720 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:48.344688    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" path="/var/lib/kubelet/pods/570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73/volumes"
Jul 28 12:47:48.626223 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:48.626143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:47:48.626554 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:48.626524    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:47:50.587921 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:50.587885    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:55.587303 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:55.587265    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:55.587673 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:55.587395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:47:58.627140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:47:58.627102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:48:00.586972 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:00.586926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:05.587215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:05.587173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:08.626836 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:08.626786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:48:10.548853 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.548802    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"]
Jul 28 12:48:10.549337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.549300    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container"
Jul 28 12:48:10.549337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.549318    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container"
Jul 28 12:48:10.549337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.549332    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kube-rbac-proxy"
Jul 28 12:48:10.549337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.549340    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kube-rbac-proxy"
Jul 28 12:48:10.549561 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.549437    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kserve-container"
Jul 28 12:48:10.549561 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.549452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="570b012f-e5f0-4b4e-b7a8-b9de5fd5cb73" containerName="kube-rbac-proxy"
Jul 28 12:48:10.556507 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.556481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:10.559155 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.559125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 12:48:10.559298 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.559125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 12:48:10.561318 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.561297    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"]
Jul 28 12:48:10.587667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.587633    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:10.699228 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.699178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82874998-e936-440e-9bde-8d6035667648-openshift-service-ca-bundle\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:10.699228 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.699236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:10.800397 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.800305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82874998-e936-440e-9bde-8d6035667648-openshift-service-ca-bundle\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:10.800397 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.800350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:10.800589 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:48:10.800496    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 28 12:48:10.800589 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:48:10.800574    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls podName:82874998-e936-440e-9bde-8d6035667648 nodeName:}" failed. No retries permitted until 2026-07-28 12:48:11.300556489 +0000 UTC m=+799.518909179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls") pod "model-chainer-58dff6d8d-9q9x9" (UID: "82874998-e936-440e-9bde-8d6035667648") : secret "model-chainer-serving-cert" not found
Jul 28 12:48:10.800991 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:10.800969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82874998-e936-440e-9bde-8d6035667648-openshift-service-ca-bundle\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:11.007171 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.007139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:48:11.102788 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.102702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9578c1e6-0abc-4744-a9e9-d1078048a18b-openshift-service-ca-bundle\") pod \"9578c1e6-0abc-4744-a9e9-d1078048a18b\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") "
Jul 28 12:48:11.102977 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.102866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls\") pod \"9578c1e6-0abc-4744-a9e9-d1078048a18b\" (UID: \"9578c1e6-0abc-4744-a9e9-d1078048a18b\") "
Jul 28 12:48:11.103161 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.103143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9578c1e6-0abc-4744-a9e9-d1078048a18b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9578c1e6-0abc-4744-a9e9-d1078048a18b" (UID: "9578c1e6-0abc-4744-a9e9-d1078048a18b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:11.105156 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.105125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9578c1e6-0abc-4744-a9e9-d1078048a18b" (UID: "9578c1e6-0abc-4744-a9e9-d1078048a18b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:48:11.204143 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.204103    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9578c1e6-0abc-4744-a9e9-d1078048a18b-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:48:11.204143 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.204137    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9578c1e6-0abc-4744-a9e9-d1078048a18b-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:48:11.305103 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.305067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:11.307670 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.307643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls\") pod \"model-chainer-58dff6d8d-9q9x9\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") " pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:11.469489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.469450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:11.610202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.610170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"]
Jul 28 12:48:11.613210 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:48:11.613186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82874998_e936_440e_9bde_8d6035667648.slice/crio-f08c3440931f4304b8dab7aabe6564bc38bb173bf4e368e1eb9ff31953bbd082 WatchSource:0}: Error finding container f08c3440931f4304b8dab7aabe6564bc38bb173bf4e368e1eb9ff31953bbd082: Status 404 returned error can't find the container with id f08c3440931f4304b8dab7aabe6564bc38bb173bf4e368e1eb9ff31953bbd082
Jul 28 12:48:11.745068 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.744979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" event={"ID":"82874998-e936-440e-9bde-8d6035667648","Type":"ContainerStarted","Data":"3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6"}
Jul 28 12:48:11.745068 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.745029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" event={"ID":"82874998-e936-440e-9bde-8d6035667648","Type":"ContainerStarted","Data":"f08c3440931f4304b8dab7aabe6564bc38bb173bf4e368e1eb9ff31953bbd082"}
Jul 28 12:48:11.745308 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.745078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:11.746210 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.746188    2563 generic.go:356] "Generic (PLEG): container finished" podID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerID="e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33" exitCode=0
Jul 28 12:48:11.746317 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.746219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"
Jul 28 12:48:11.746317 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.746239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" event={"ID":"9578c1e6-0abc-4744-a9e9-d1078048a18b","Type":"ContainerDied","Data":"e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33"}
Jul 28 12:48:11.746317 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.746279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx" event={"ID":"9578c1e6-0abc-4744-a9e9-d1078048a18b","Type":"ContainerDied","Data":"5695caa1107b773a4640b8c5fcaac16dd5d9556f47aa49fb40a1a4568fc77a2a"}
Jul 28 12:48:11.746317 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.746299    2563 scope.go:117] "RemoveContainer" containerID="e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33"
Jul 28 12:48:11.755061 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.755036    2563 scope.go:117] "RemoveContainer" containerID="e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33"
Jul 28 12:48:11.755336 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:48:11.755316    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33\": container with ID starting with e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33 not found: ID does not exist" containerID="e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33"
Jul 28 12:48:11.755406 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.755351    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33"} err="failed to get container status \"e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33\": rpc error: code = NotFound desc = could not find container \"e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33\": container with ID starting with e46166113d8f6709dff93bad49c77f46943d58f9c4e23dbf1665ed9384bc4e33 not found: ID does not exist"
Jul 28 12:48:11.764852 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.764786    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podStartSLOduration=1.764770402 podStartE2EDuration="1.764770402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:48:11.763024965 +0000 UTC m=+799.981377677" watchObservedRunningTime="2026-07-28 12:48:11.764770402 +0000 UTC m=+799.983123113"
Jul 28 12:48:11.781013 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.780981    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"]
Jul 28 12:48:11.783101 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:11.783074    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9baeb-7554d76cc8-9rcpx"]
Jul 28 12:48:12.344250 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:12.344219    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" path="/var/lib/kubelet/pods/9578c1e6-0abc-4744-a9e9-d1078048a18b/volumes"
Jul 28 12:48:17.763161 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:17.763081    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:18.627535 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:18.627499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:48:20.677249 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.677215    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"]
Jul 28 12:48:20.677711 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.677421    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" containerID="cri-o://3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6" gracePeriod=30
Jul 28 12:48:20.984299 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.984263    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"]
Jul 28 12:48:20.984631 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.984618    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb"
Jul 28 12:48:20.984678 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.984634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb"
Jul 28 12:48:20.984713 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.984691    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9578c1e6-0abc-4744-a9e9-d1078048a18b" containerName="switch-graph-9baeb"
Jul 28 12:48:20.987804 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.987780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:20.990023 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.989990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7fa9-predictor-serving-cert\""
Jul 28 12:48:20.990276 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.990259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7fa9-kube-rbac-proxy-sar-config\""
Jul 28 12:48:20.998656 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:20.998632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"]
Jul 28 12:48:21.088345 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.088306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8z7\" (UniqueName: \"kubernetes.io/projected/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-api-access-2k8z7\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.088532 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.088357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.088532 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.088386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.189643 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.189609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8z7\" (UniqueName: \"kubernetes.io/projected/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-api-access-2k8z7\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.189808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.189658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.189808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.189700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.190023 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:48:21.189862    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-serving-cert: secret "error-404-isvc-f7fa9-predictor-serving-cert" not found
Jul 28 12:48:21.190023 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:48:21.189939    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls podName:40f65ed7-cd19-4d30-bdb0-a94bb80d36d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:48:21.68992295 +0000 UTC m=+809.908275641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls") pod "error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" (UID: "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0") : secret "error-404-isvc-f7fa9-predictor-serving-cert" not found
Jul 28 12:48:21.190416 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.190396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.200870 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.200843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8z7\" (UniqueName: \"kubernetes.io/projected/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-api-access-2k8z7\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.694358 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.694323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.696911 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.696891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls\") pod \"error-404-isvc-f7fa9-predictor-7f68c498d-b2czf\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") " pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:21.898275 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:21.898236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:22.046808 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.046768    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"]
Jul 28 12:48:22.050433 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:48:22.050401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f65ed7_cd19_4d30_bdb0_a94bb80d36d0.slice/crio-909b2c601cd9883f65274e5f39594f26ca8593248597cd1d7586e22acfe393e7 WatchSource:0}: Error finding container 909b2c601cd9883f65274e5f39594f26ca8593248597cd1d7586e22acfe393e7: Status 404 returned error can't find the container with id 909b2c601cd9883f65274e5f39594f26ca8593248597cd1d7586e22acfe393e7
Jul 28 12:48:22.761437 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.761400    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:22.805023 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.804985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" event={"ID":"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0","Type":"ContainerStarted","Data":"30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1"}
Jul 28 12:48:22.805231 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.805040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" event={"ID":"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0","Type":"ContainerStarted","Data":"0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264"}
Jul 28 12:48:22.805231 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.805062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" event={"ID":"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0","Type":"ContainerStarted","Data":"909b2c601cd9883f65274e5f39594f26ca8593248597cd1d7586e22acfe393e7"}
Jul 28 12:48:22.805231 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.805136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:22.823292 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:22.823226    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podStartSLOduration=2.823209526 podStartE2EDuration="2.823209526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:48:22.82231325 +0000 UTC m=+811.040665976" watchObservedRunningTime="2026-07-28 12:48:22.823209526 +0000 UTC m=+811.041562239"
Jul 28 12:48:23.809679 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:23.809650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:23.811185 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:23.811154    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 12:48:24.815048 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:24.815009    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 12:48:27.761423 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:27.761385    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:28.628032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:28.627999    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:48:29.819705 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:29.819677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:48:29.820178 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:29.820153    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 12:48:32.761914 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:32.761871    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:32.762415 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:32.761977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:37.761460 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:37.761419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:39.821182 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:39.821140    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 12:48:41.117609 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.117556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"]
Jul 28 12:48:41.121211 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.121194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.123534 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.123508    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-fc405-kube-rbac-proxy-sar-config\""
Jul 28 12:48:41.123534 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.123508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-fc405-serving-cert\""
Jul 28 12:48:41.130635 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.130607    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"]
Jul 28 12:48:41.159088 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.159054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0510bf3d-66a3-4f0f-8036-55880df89546-proxy-tls\") pod \"switch-graph-fc405-8559d54778-7t8j8\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") " pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.159264 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.159108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0510bf3d-66a3-4f0f-8036-55880df89546-openshift-service-ca-bundle\") pod \"switch-graph-fc405-8559d54778-7t8j8\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") " pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.260558 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.260520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0510bf3d-66a3-4f0f-8036-55880df89546-openshift-service-ca-bundle\") pod \"switch-graph-fc405-8559d54778-7t8j8\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") " pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.260749 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.260611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0510bf3d-66a3-4f0f-8036-55880df89546-proxy-tls\") pod \"switch-graph-fc405-8559d54778-7t8j8\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") " pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.261273 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.261245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0510bf3d-66a3-4f0f-8036-55880df89546-openshift-service-ca-bundle\") pod \"switch-graph-fc405-8559d54778-7t8j8\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") " pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.263357 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.263331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0510bf3d-66a3-4f0f-8036-55880df89546-proxy-tls\") pod \"switch-graph-fc405-8559d54778-7t8j8\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") " pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.433279 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.433169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.577619 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.577587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"]
Jul 28 12:48:41.580927 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:48:41.580895    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0510bf3d_66a3_4f0f_8036_55880df89546.slice/crio-ef2b1c78485e6db928234936dc3d15ded1d427aa4540ef4c1ee583d99b957277 WatchSource:0}: Error finding container ef2b1c78485e6db928234936dc3d15ded1d427aa4540ef4c1ee583d99b957277: Status 404 returned error can't find the container with id ef2b1c78485e6db928234936dc3d15ded1d427aa4540ef4c1ee583d99b957277
Jul 28 12:48:41.892253 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.892211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" event={"ID":"0510bf3d-66a3-4f0f-8036-55880df89546","Type":"ContainerStarted","Data":"ea835b3eebd7027aaee30cc138804210a1866cb04ede3503dd633cf327f1f19e"}
Jul 28 12:48:41.892438 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.892264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" event={"ID":"0510bf3d-66a3-4f0f-8036-55880df89546","Type":"ContainerStarted","Data":"ef2b1c78485e6db928234936dc3d15ded1d427aa4540ef4c1ee583d99b957277"}
Jul 28 12:48:41.892438 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.892346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:41.910803 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:41.910745    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podStartSLOduration=0.910728233 podStartE2EDuration="910.728233ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:48:41.910499987 +0000 UTC m=+830.128852702" watchObservedRunningTime="2026-07-28 12:48:41.910728233 +0000 UTC m=+830.129080944"
Jul 28 12:48:42.761694 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:42.761655    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:47.761630 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:47.761583    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:47.903021 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:47.902989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:48:49.820199 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:49.820161    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 12:48:50.837913 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.837884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:50.933475 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.933378    2563 generic.go:356] "Generic (PLEG): container finished" podID="82874998-e936-440e-9bde-8d6035667648" containerID="3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6" exitCode=0
Jul 28 12:48:50.933475 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.933436    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"
Jul 28 12:48:50.933703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.933439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" event={"ID":"82874998-e936-440e-9bde-8d6035667648","Type":"ContainerDied","Data":"3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6"}
Jul 28 12:48:50.933703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.933561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9" event={"ID":"82874998-e936-440e-9bde-8d6035667648","Type":"ContainerDied","Data":"f08c3440931f4304b8dab7aabe6564bc38bb173bf4e368e1eb9ff31953bbd082"}
Jul 28 12:48:50.933703 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.933585    2563 scope.go:117] "RemoveContainer" containerID="3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6"
Jul 28 12:48:50.942233 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.942211    2563 scope.go:117] "RemoveContainer" containerID="3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6"
Jul 28 12:48:50.942539 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:48:50.942519    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6\": container with ID starting with 3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6 not found: ID does not exist" containerID="3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6"
Jul 28 12:48:50.942599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.942551    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6"} err="failed to get container status \"3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6\": rpc error: code = NotFound desc = could not find container \"3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6\": container with ID starting with 3467873d030436bf9ee6bd4fa59215d9c266ed6554d9de7fba42a23fd0ef57e6 not found: ID does not exist"
Jul 28 12:48:50.944865 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.944848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls\") pod \"82874998-e936-440e-9bde-8d6035667648\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") "
Jul 28 12:48:50.944929 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.944900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82874998-e936-440e-9bde-8d6035667648-openshift-service-ca-bundle\") pod \"82874998-e936-440e-9bde-8d6035667648\" (UID: \"82874998-e936-440e-9bde-8d6035667648\") "
Jul 28 12:48:50.945228 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.945205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82874998-e936-440e-9bde-8d6035667648-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "82874998-e936-440e-9bde-8d6035667648" (UID: "82874998-e936-440e-9bde-8d6035667648"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:50.947023 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:50.947000    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82874998-e936-440e-9bde-8d6035667648" (UID: "82874998-e936-440e-9bde-8d6035667648"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:48:51.045744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:51.045702    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82874998-e936-440e-9bde-8d6035667648-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:48:51.045744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:51.045738    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82874998-e936-440e-9bde-8d6035667648-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:48:51.255428 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:51.255356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"]
Jul 28 12:48:51.257236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:51.257212    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-58dff6d8d-9q9x9"]
Jul 28 12:48:52.346405 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:52.346375    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82874998-e936-440e-9bde-8d6035667648" path="/var/lib/kubelet/pods/82874998-e936-440e-9bde-8d6035667648/volumes"
Jul 28 12:48:59.821013 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:48:59.820964    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 12:49:09.821918 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:09.821883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:49:30.908089 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.908044    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"]
Jul 28 12:49:30.908966 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.908940    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer"
Jul 28 12:49:30.908966 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.908968    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer"
Jul 28 12:49:30.909148 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.909135    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82874998-e936-440e-9bde-8d6035667648" containerName="model-chainer"
Jul 28 12:49:30.913142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.913114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:30.915546 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.915521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f7fa9-kube-rbac-proxy-sar-config\""
Jul 28 12:49:30.915658 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.915524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f7fa9-serving-cert\""
Jul 28 12:49:30.922565 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.922528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"]
Jul 28 12:49:30.973307 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.973268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-openshift-service-ca-bundle\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:30.973489 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:30.973335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.074447 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.074391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-openshift-service-ca-bundle\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.074629 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.074465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.074629 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:49:31.074573    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-f7fa9-serving-cert: secret "sequence-graph-f7fa9-serving-cert" not found
Jul 28 12:49:31.074713 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:49:31.074639    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls podName:6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f nodeName:}" failed. No retries permitted until 2026-07-28 12:49:31.574622281 +0000 UTC m=+879.792974972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls") pod "sequence-graph-f7fa9-649ffd79b6-dt6lz" (UID: "6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f") : secret "sequence-graph-f7fa9-serving-cert" not found
Jul 28 12:49:31.075090 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.075068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-openshift-service-ca-bundle\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.579105 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.579062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.581613 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.581589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls\") pod \"sequence-graph-f7fa9-649ffd79b6-dt6lz\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") " pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.823568 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.823532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:31.961971 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:31.961942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"]
Jul 28 12:49:31.964419 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:49:31.964390    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6565bb02_bd4e_4b97_ab06_fbf3ce9b6f5f.slice/crio-d9e6dcf8892834962e9b1edc4d8d5a464cc6763ffc3cda774f186ff988b177e4 WatchSource:0}: Error finding container d9e6dcf8892834962e9b1edc4d8d5a464cc6763ffc3cda774f186ff988b177e4: Status 404 returned error can't find the container with id d9e6dcf8892834962e9b1edc4d8d5a464cc6763ffc3cda774f186ff988b177e4
Jul 28 12:49:32.118461 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:32.118357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" event={"ID":"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f","Type":"ContainerStarted","Data":"a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128"}
Jul 28 12:49:32.118461 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:32.118412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" event={"ID":"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f","Type":"ContainerStarted","Data":"d9e6dcf8892834962e9b1edc4d8d5a464cc6763ffc3cda774f186ff988b177e4"}
Jul 28 12:49:32.118461 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:32.118448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:32.136643 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:32.136587    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podStartSLOduration=2.136571287 podStartE2EDuration="2.136571287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:49:32.135711724 +0000 UTC m=+880.354064436" watchObservedRunningTime="2026-07-28 12:49:32.136571287 +0000 UTC m=+880.354924019"
Jul 28 12:49:38.128868 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:38.128830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:49:52.275868 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:52.275769    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:49:52.276444 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:49:52.276301    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:54:52.309179 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:54:52.309152    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:54:52.309762 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:54:52.309746    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:56:55.782424 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:55.782389    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"]
Jul 28 12:56:55.785055 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:55.782665    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" containerID="cri-o://ea835b3eebd7027aaee30cc138804210a1866cb04ede3503dd633cf327f1f19e" gracePeriod=30
Jul 28 12:56:56.042205 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.042120    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"]
Jul 28 12:56:56.042371 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.042347    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" containerID="cri-o://7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4" gracePeriod=30
Jul 28 12:56:56.042448 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.042392    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kube-rbac-proxy" containerID="cri-o://8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6" gracePeriod=30
Jul 28 12:56:56.043621 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.043599    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"]
Jul 28 12:56:56.047474 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.047045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.051531 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.051510    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-75e21-predictor-serving-cert\""
Jul 28 12:56:56.051641 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.051596    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-75e21-kube-rbac-proxy-sar-config\""
Jul 28 12:56:56.060670 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.060649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"]
Jul 28 12:56:56.218225 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.218190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qp7z\" (UniqueName: \"kubernetes.io/projected/9ef29225-a748-4be2-8e05-af974a483507-kube-api-access-9qp7z\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.218373 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.218232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ef29225-a748-4be2-8e05-af974a483507-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.218373 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.218258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.319744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.319662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qp7z\" (UniqueName: \"kubernetes.io/projected/9ef29225-a748-4be2-8e05-af974a483507-kube-api-access-9qp7z\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.319744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.319701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ef29225-a748-4be2-8e05-af974a483507-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.319744 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.319733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.320035 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:56:56.319858    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-75e21-predictor-serving-cert: secret "error-404-isvc-75e21-predictor-serving-cert" not found
Jul 28 12:56:56.320035 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:56:56.319919    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls podName:9ef29225-a748-4be2-8e05-af974a483507 nodeName:}" failed. No retries permitted until 2026-07-28 12:56:56.81989787 +0000 UTC m=+1325.038250559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls") pod "error-404-isvc-75e21-predictor-64646f5b4b-6j67r" (UID: "9ef29225-a748-4be2-8e05-af974a483507") : secret "error-404-isvc-75e21-predictor-serving-cert" not found
Jul 28 12:56:56.320375 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.320355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ef29225-a748-4be2-8e05-af974a483507-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.328562 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.328543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qp7z\" (UniqueName: \"kubernetes.io/projected/9ef29225-a748-4be2-8e05-af974a483507-kube-api-access-9qp7z\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.825244 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.825211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.827665 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.827644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls\") pod \"error-404-isvc-75e21-predictor-64646f5b4b-6j67r\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") " pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:56.966142 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:56.966109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:57.068083 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:57.067872    2563 generic.go:356] "Generic (PLEG): container finished" podID="377a0412-19a2-43c5-a249-a7edecac9337" containerID="8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6" exitCode=2
Jul 28 12:56:57.068083 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:57.068002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" event={"ID":"377a0412-19a2-43c5-a249-a7edecac9337","Type":"ContainerDied","Data":"8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6"}
Jul 28 12:56:57.100595 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:57.100516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"]
Jul 28 12:56:57.103407 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:56:57.103373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef29225_a748_4be2_8e05_af974a483507.slice/crio-bd35fa893d2fecf9d851c861ca0dde6d576ce1b877456da4251d5ab258eae0ea WatchSource:0}: Error finding container bd35fa893d2fecf9d851c861ca0dde6d576ce1b877456da4251d5ab258eae0ea: Status 404 returned error can't find the container with id bd35fa893d2fecf9d851c861ca0dde6d576ce1b877456da4251d5ab258eae0ea
Jul 28 12:56:57.901242 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:57.901204    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:56:58.075806 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.075768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" event={"ID":"9ef29225-a748-4be2-8e05-af974a483507","Type":"ContainerStarted","Data":"f7457a37c113d9c295ec6206303e44de798480881a3f47126fc38c558890a7eb"}
Jul 28 12:56:58.075982 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.075859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:58.075982 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.075882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" event={"ID":"9ef29225-a748-4be2-8e05-af974a483507","Type":"ContainerStarted","Data":"c2bb77740961941c08220939557ac2210895b5759840588e9939d91aaa936597"}
Jul 28 12:56:58.075982 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.075900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" event={"ID":"9ef29225-a748-4be2-8e05-af974a483507","Type":"ContainerStarted","Data":"bd35fa893d2fecf9d851c861ca0dde6d576ce1b877456da4251d5ab258eae0ea"}
Jul 28 12:56:58.096383 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.096332    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podStartSLOduration=2.096318516 podStartE2EDuration="2.096318516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:58.094894316 +0000 UTC m=+1326.313247029" watchObservedRunningTime="2026-07-28 12:56:58.096318516 +0000 UTC m=+1326.314671227"
Jul 28 12:56:58.621321 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.621276    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 12:56:58.626681 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:58.626654    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 12:56:59.080413 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.080383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:56:59.081778 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.081751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 12:56:59.291918 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.291890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:56:59.448516 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.448486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45r9s\" (UniqueName: \"kubernetes.io/projected/377a0412-19a2-43c5-a249-a7edecac9337-kube-api-access-45r9s\") pod \"377a0412-19a2-43c5-a249-a7edecac9337\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") "
Jul 28 12:56:59.448668 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.448551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/377a0412-19a2-43c5-a249-a7edecac9337-kube-rbac-proxy-sar-config\") pod \"377a0412-19a2-43c5-a249-a7edecac9337\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") "
Jul 28 12:56:59.448668 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.448609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls\") pod \"377a0412-19a2-43c5-a249-a7edecac9337\" (UID: \"377a0412-19a2-43c5-a249-a7edecac9337\") "
Jul 28 12:56:59.449006 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.448971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377a0412-19a2-43c5-a249-a7edecac9337-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "377a0412-19a2-43c5-a249-a7edecac9337" (UID: "377a0412-19a2-43c5-a249-a7edecac9337"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:59.450856 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.450835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377a0412-19a2-43c5-a249-a7edecac9337-kube-api-access-45r9s" (OuterVolumeSpecName: "kube-api-access-45r9s") pod "377a0412-19a2-43c5-a249-a7edecac9337" (UID: "377a0412-19a2-43c5-a249-a7edecac9337"). InnerVolumeSpecName "kube-api-access-45r9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:59.450856 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.450844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "377a0412-19a2-43c5-a249-a7edecac9337" (UID: "377a0412-19a2-43c5-a249-a7edecac9337"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:59.550174 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.550137    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/377a0412-19a2-43c5-a249-a7edecac9337-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:56:59.550174 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.550168    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45r9s\" (UniqueName: \"kubernetes.io/projected/377a0412-19a2-43c5-a249-a7edecac9337-kube-api-access-45r9s\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:56:59.550174 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:56:59.550177    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/377a0412-19a2-43c5-a249-a7edecac9337-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:57:00.086227 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.086192    2563 generic.go:356] "Generic (PLEG): container finished" podID="377a0412-19a2-43c5-a249-a7edecac9337" containerID="7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4" exitCode=0
Jul 28 12:57:00.086710 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.086249    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"
Jul 28 12:57:00.086710 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.086242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" event={"ID":"377a0412-19a2-43c5-a249-a7edecac9337","Type":"ContainerDied","Data":"7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4"}
Jul 28 12:57:00.086710 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.086395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg" event={"ID":"377a0412-19a2-43c5-a249-a7edecac9337","Type":"ContainerDied","Data":"c5c0e7867dff585d5b8acba0bc9dafdb3d9f44adffe0f98c06b039b9705160c3"}
Jul 28 12:57:00.086710 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.086424    2563 scope.go:117] "RemoveContainer" containerID="8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6"
Jul 28 12:57:00.087038 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.086891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 12:57:00.095742 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.095726    2563 scope.go:117] "RemoveContainer" containerID="7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4"
Jul 28 12:57:00.103651 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.103635    2563 scope.go:117] "RemoveContainer" containerID="8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6"
Jul 28 12:57:00.103953 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:00.103934    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6\": container with ID starting with 8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6 not found: ID does not exist" containerID="8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6"
Jul 28 12:57:00.104019 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.103961    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6"} err="failed to get container status \"8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6\": rpc error: code = NotFound desc = could not find container \"8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6\": container with ID starting with 8979fdc149d3675a563e702e4b5445692803c976a33a994f2a7852decdbe10a6 not found: ID does not exist"
Jul 28 12:57:00.104019 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.103979    2563 scope.go:117] "RemoveContainer" containerID="7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4"
Jul 28 12:57:00.104236 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:00.104219    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4\": container with ID starting with 7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4 not found: ID does not exist" containerID="7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4"
Jul 28 12:57:00.104287 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.104243    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4"} err="failed to get container status \"7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4\": rpc error: code = NotFound desc = could not find container \"7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4\": container with ID starting with 7e89f25eb988affbc71a68dc0fc2808c06712b8db83afb280bcf9c31d4a2a0d4 not found: ID does not exist"
Jul 28 12:57:00.114848 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.114809    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"]
Jul 28 12:57:00.118949 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.118927    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc405-predictor-86dddcdd4b-hh8lg"]
Jul 28 12:57:00.344342 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:00.344250    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377a0412-19a2-43c5-a249-a7edecac9337" path="/var/lib/kubelet/pods/377a0412-19a2-43c5-a249-a7edecac9337/volumes"
Jul 28 12:57:02.901306 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:02.901269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:05.091797 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:05.091764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:57:05.092286 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:05.092191    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 12:57:07.900917 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:07.900866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:07.901290 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:07.900993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:57:12.900532 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:12.900491    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:15.092410 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:15.092322    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 12:57:17.901081 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:17.901047    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:22.900545 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:22.900500    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:25.093157 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:25.093115    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 12:57:26.209366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.209331    2563 generic.go:356] "Generic (PLEG): container finished" podID="0510bf3d-66a3-4f0f-8036-55880df89546" containerID="ea835b3eebd7027aaee30cc138804210a1866cb04ede3503dd633cf327f1f19e" exitCode=0
Jul 28 12:57:26.209731 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.209376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" event={"ID":"0510bf3d-66a3-4f0f-8036-55880df89546","Type":"ContainerDied","Data":"ea835b3eebd7027aaee30cc138804210a1866cb04ede3503dd633cf327f1f19e"}
Jul 28 12:57:26.425343 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.425317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:57:26.461112 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.461029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0510bf3d-66a3-4f0f-8036-55880df89546-openshift-service-ca-bundle\") pod \"0510bf3d-66a3-4f0f-8036-55880df89546\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") "
Jul 28 12:57:26.461112 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.461074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0510bf3d-66a3-4f0f-8036-55880df89546-proxy-tls\") pod \"0510bf3d-66a3-4f0f-8036-55880df89546\" (UID: \"0510bf3d-66a3-4f0f-8036-55880df89546\") "
Jul 28 12:57:26.461427 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.461402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0510bf3d-66a3-4f0f-8036-55880df89546-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0510bf3d-66a3-4f0f-8036-55880df89546" (UID: "0510bf3d-66a3-4f0f-8036-55880df89546"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:26.463322 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.463293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0510bf3d-66a3-4f0f-8036-55880df89546-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0510bf3d-66a3-4f0f-8036-55880df89546" (UID: "0510bf3d-66a3-4f0f-8036-55880df89546"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:26.562023 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.561977    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0510bf3d-66a3-4f0f-8036-55880df89546-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:57:26.562023 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:26.562006    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0510bf3d-66a3-4f0f-8036-55880df89546-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:57:27.214499 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:27.214468    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"
Jul 28 12:57:27.215052 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:27.214499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8" event={"ID":"0510bf3d-66a3-4f0f-8036-55880df89546","Type":"ContainerDied","Data":"ef2b1c78485e6db928234936dc3d15ded1d427aa4540ef4c1ee583d99b957277"}
Jul 28 12:57:27.215052 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:27.214552    2563 scope.go:117] "RemoveContainer" containerID="ea835b3eebd7027aaee30cc138804210a1866cb04ede3503dd633cf327f1f19e"
Jul 28 12:57:27.242207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:27.242177    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"]
Jul 28 12:57:27.258674 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:27.258650    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-fc405-8559d54778-7t8j8"]
Jul 28 12:57:28.344926 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:28.344898    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" path="/var/lib/kubelet/pods/0510bf3d-66a3-4f0f-8036-55880df89546/volumes"
Jul 28 12:57:35.092278 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:35.092241    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 12:57:45.094012 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.093974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:57:45.604576 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.604544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"]
Jul 28 12:57:45.604799 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.604775    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" containerID="cri-o://a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128" gracePeriod=30
Jul 28 12:57:45.753368 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.753335    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"]
Jul 28 12:57:45.753608 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.753557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container" containerID="cri-o://0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264" gracePeriod=30
Jul 28 12:57:45.753697 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.753598    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kube-rbac-proxy" containerID="cri-o://30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1" gracePeriod=30
Jul 28 12:57:45.854640 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.854556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"]
Jul 28 12:57:45.855003 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.854985    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container"
Jul 28 12:57:45.855102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855005    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container"
Jul 28 12:57:45.855102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855040    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kube-rbac-proxy"
Jul 28 12:57:45.855102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855048    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kube-rbac-proxy"
Jul 28 12:57:45.855102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855060    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405"
Jul 28 12:57:45.855102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855069    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405"
Jul 28 12:57:45.855356 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855150    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0510bf3d-66a3-4f0f-8036-55880df89546" containerName="switch-graph-fc405"
Jul 28 12:57:45.855356 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855163    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kserve-container"
Jul 28 12:57:45.855356 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.855177    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="377a0412-19a2-43c5-a249-a7edecac9337" containerName="kube-rbac-proxy"
Jul 28 12:57:45.859681 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.859661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:45.862291 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.862268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fc1de-predictor-serving-cert\""
Jul 28 12:57:45.862400 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.862275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fc1de-kube-rbac-proxy-sar-config\""
Jul 28 12:57:45.871929 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.871910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"]
Jul 28 12:57:45.922567 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.922539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htp55\" (UniqueName: \"kubernetes.io/projected/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-api-access-htp55\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:45.922707 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.922576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:45.922707 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:45.922673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.023915 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.023875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.024099 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.023931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htp55\" (UniqueName: \"kubernetes.io/projected/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-api-access-htp55\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.024099 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.023961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.024099 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:46.024076    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-serving-cert: secret "error-404-isvc-fc1de-predictor-serving-cert" not found
Jul 28 12:57:46.024210 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:46.024153    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls podName:12056daa-b8ce-4537-b4dd-626a61607ecb nodeName:}" failed. No retries permitted until 2026-07-28 12:57:46.524134212 +0000 UTC m=+1374.742486901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls") pod "error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" (UID: "12056daa-b8ce-4537-b4dd-626a61607ecb") : secret "error-404-isvc-fc1de-predictor-serving-cert" not found
Jul 28 12:57:46.024592 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.024570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.036032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.036008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htp55\" (UniqueName: \"kubernetes.io/projected/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-api-access-htp55\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.300616 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.300585    2563 generic.go:356] "Generic (PLEG): container finished" podID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerID="30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1" exitCode=2
Jul 28 12:57:46.300997 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.300637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" event={"ID":"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0","Type":"ContainerDied","Data":"30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1"}
Jul 28 12:57:46.527650 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:46.527617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:46.527811 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:46.527722    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-serving-cert: secret "error-404-isvc-fc1de-predictor-serving-cert" not found
Jul 28 12:57:46.527811 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:46.527773    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls podName:12056daa-b8ce-4537-b4dd-626a61607ecb nodeName:}" failed. No retries permitted until 2026-07-28 12:57:47.527759615 +0000 UTC m=+1375.746112306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls") pod "error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" (UID: "12056daa-b8ce-4537-b4dd-626a61607ecb") : secret "error-404-isvc-fc1de-predictor-serving-cert" not found
Jul 28 12:57:47.536235 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:47.536189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:47.538751 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:47.538726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") pod \"error-404-isvc-fc1de-predictor-6bc584d948-4fgbk\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") " pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:47.670228 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:47.670190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:47.801964 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:47.801941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"]
Jul 28 12:57:47.804099 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:57:47.804072    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12056daa_b8ce_4537_b4dd_626a61607ecb.slice/crio-22c7128da742fd2a341d9beca18673469ec0ec70db23665195910647f9ef9d3a WatchSource:0}: Error finding container 22c7128da742fd2a341d9beca18673469ec0ec70db23665195910647f9ef9d3a: Status 404 returned error can't find the container with id 22c7128da742fd2a341d9beca18673469ec0ec70db23665195910647f9ef9d3a
Jul 28 12:57:48.127188 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.127085    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:48.314595 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.314563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" event={"ID":"12056daa-b8ce-4537-b4dd-626a61607ecb","Type":"ContainerStarted","Data":"a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758"}
Jul 28 12:57:48.314790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.314627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" event={"ID":"12056daa-b8ce-4537-b4dd-626a61607ecb","Type":"ContainerStarted","Data":"4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933"}
Jul 28 12:57:48.314790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.314655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" event={"ID":"12056daa-b8ce-4537-b4dd-626a61607ecb","Type":"ContainerStarted","Data":"22c7128da742fd2a341d9beca18673469ec0ec70db23665195910647f9ef9d3a"}
Jul 28 12:57:48.314790 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.314678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:48.334316 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.334274    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podStartSLOduration=3.3342558159999998 podStartE2EDuration="3.334255816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:57:48.333974532 +0000 UTC m=+1376.552327246" watchObservedRunningTime="2026-07-28 12:57:48.334255816 +0000 UTC m=+1376.552608526"
Jul 28 12:57:48.915300 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:48.915272    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:57:49.047927 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.047888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k8z7\" (UniqueName: \"kubernetes.io/projected/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-api-access-2k8z7\") pod \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") "
Jul 28 12:57:49.048124 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.047977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls\") pod \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") "
Jul 28 12:57:49.048124 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.048022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-rbac-proxy-sar-config\") pod \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\" (UID: \"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0\") "
Jul 28 12:57:49.048393 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.048360    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" (UID: "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:49.050257 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.050069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-api-access-2k8z7" (OuterVolumeSpecName: "kube-api-access-2k8z7") pod "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" (UID: "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0"). InnerVolumeSpecName "kube-api-access-2k8z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:57:49.060040 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.050259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" (UID: "40f65ed7-cd19-4d30-bdb0-a94bb80d36d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:49.148966 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.148935    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:57:49.148966 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.148962    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:57:49.148966 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.148971    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8z7\" (UniqueName: \"kubernetes.io/projected/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0-kube-api-access-2k8z7\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:57:49.321217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.321135    2563 generic.go:356] "Generic (PLEG): container finished" podID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerID="0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264" exitCode=0
Jul 28 12:57:49.321217 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.321197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"
Jul 28 12:57:49.321599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.321216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" event={"ID":"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0","Type":"ContainerDied","Data":"0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264"}
Jul 28 12:57:49.321599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.321299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf" event={"ID":"40f65ed7-cd19-4d30-bdb0-a94bb80d36d0","Type":"ContainerDied","Data":"909b2c601cd9883f65274e5f39594f26ca8593248597cd1d7586e22acfe393e7"}
Jul 28 12:57:49.321599 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.321327    2563 scope.go:117] "RemoveContainer" containerID="30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1"
Jul 28 12:57:49.321876 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.321791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:49.323314 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.323290    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 12:57:49.330273 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.330259    2563 scope.go:117] "RemoveContainer" containerID="0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264"
Jul 28 12:57:49.337522 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.337506    2563 scope.go:117] "RemoveContainer" containerID="30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1"
Jul 28 12:57:49.337768 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:49.337750    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1\": container with ID starting with 30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1 not found: ID does not exist" containerID="30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1"
Jul 28 12:57:49.337855 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.337775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1"} err="failed to get container status \"30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1\": rpc error: code = NotFound desc = could not find container \"30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1\": container with ID starting with 30f48c4bf76cb1cebd891630c1d4f7e77ade71a0fe196f3e7fd8eec37d6c6ff1 not found: ID does not exist"
Jul 28 12:57:49.337855 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.337792    2563 scope.go:117] "RemoveContainer" containerID="0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264"
Jul 28 12:57:49.338031 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:49.338017    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264\": container with ID starting with 0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264 not found: ID does not exist" containerID="0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264"
Jul 28 12:57:49.338081 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.338036    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264"} err="failed to get container status \"0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264\": rpc error: code = NotFound desc = could not find container \"0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264\": container with ID starting with 0218cfe89b3f9f577044594fde1b9e81b8a37dc80f2bfc2a2db71daf5775e264 not found: ID does not exist"
Jul 28 12:57:49.342597 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.342576    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"]
Jul 28 12:57:49.349017 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:49.348998    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7fa9-predictor-7f68c498d-b2czf"]
Jul 28 12:57:50.327340 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:50.327300    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 12:57:50.344413 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:50.344385    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" path="/var/lib/kubelet/pods/40f65ed7-cd19-4d30-bdb0-a94bb80d36d0/volumes"
Jul 28 12:57:53.127843 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:53.127797    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:55.332384 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:55.332359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:57:55.332890 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:55.332860    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 12:57:56.022875 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.022836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"]
Jul 28 12:57:56.023300 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.023280    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container"
Jul 28 12:57:56.023388 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.023302    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container"
Jul 28 12:57:56.023388 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.023318    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kube-rbac-proxy"
Jul 28 12:57:56.023388 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.023326    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kube-rbac-proxy"
Jul 28 12:57:56.023535 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.023413    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kserve-container"
Jul 28 12:57:56.023535 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.023427    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f65ed7-cd19-4d30-bdb0-a94bb80d36d0" containerName="kube-rbac-proxy"
Jul 28 12:57:56.027776 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.027757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.030061 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.030037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-75e21-serving-cert\""
Jul 28 12:57:56.030171 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.030080    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-75e21-kube-rbac-proxy-sar-config\""
Jul 28 12:57:56.035020 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.035001    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"]
Jul 28 12:57:56.108301 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.108270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.108459 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.108305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92af120-de57-4b4d-8420-088863f54fe2-openshift-service-ca-bundle\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.209716 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.209686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.209875 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.209725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92af120-de57-4b4d-8420-088863f54fe2-openshift-service-ca-bundle\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.209875 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:56.209859    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-75e21-serving-cert: secret "ensemble-graph-75e21-serving-cert" not found
Jul 28 12:57:56.209946 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:57:56.209940    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls podName:c92af120-de57-4b4d-8420-088863f54fe2 nodeName:}" failed. No retries permitted until 2026-07-28 12:57:56.709922087 +0000 UTC m=+1384.928274777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls") pod "ensemble-graph-75e21-5bdd974b9c-4h42x" (UID: "c92af120-de57-4b4d-8420-088863f54fe2") : secret "ensemble-graph-75e21-serving-cert" not found
Jul 28 12:57:56.210337 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.210319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92af120-de57-4b4d-8420-088863f54fe2-openshift-service-ca-bundle\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.713318 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.713281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.715930 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.715907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls\") pod \"ensemble-graph-75e21-5bdd974b9c-4h42x\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") " pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:56.938654 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:56.938624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:57.072678 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:57.072655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"]
Jul 28 12:57:57.075110 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:57:57.075081    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92af120_de57_4b4d_8420_088863f54fe2.slice/crio-e99996034aa0e9a69c314f88aae1209282bbf518f9ad6d518b8b1ce3f1519916 WatchSource:0}: Error finding container e99996034aa0e9a69c314f88aae1209282bbf518f9ad6d518b8b1ce3f1519916: Status 404 returned error can't find the container with id e99996034aa0e9a69c314f88aae1209282bbf518f9ad6d518b8b1ce3f1519916
Jul 28 12:57:57.367686 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:57.367589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" event={"ID":"c92af120-de57-4b4d-8420-088863f54fe2","Type":"ContainerStarted","Data":"d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a"}
Jul 28 12:57:57.367686 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:57.367640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" event={"ID":"c92af120-de57-4b4d-8420-088863f54fe2","Type":"ContainerStarted","Data":"e99996034aa0e9a69c314f88aae1209282bbf518f9ad6d518b8b1ce3f1519916"}
Jul 28 12:57:57.367686 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:57.367671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:57:57.387616 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:57.387564    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podStartSLOduration=1.387546194 podStartE2EDuration="1.387546194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:57:57.385541856 +0000 UTC m=+1385.603894569" watchObservedRunningTime="2026-07-28 12:57:57.387546194 +0000 UTC m=+1385.605898906"
Jul 28 12:57:58.127797 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:58.127755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:57:58.128192 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:57:58.127886    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:58:03.127151 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:03.127105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:03.377977 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:03.377898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:58:05.332846 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:05.332794    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 12:58:06.107669 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.107584    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"]
Jul 28 12:58:06.107806 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.107752    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" containerID="cri-o://d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a" gracePeriod=30
Jul 28 12:58:06.251021 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.250990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"]
Jul 28 12:58:06.251237 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.251217    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container" containerID="cri-o://c2bb77740961941c08220939557ac2210895b5759840588e9939d91aaa936597" gracePeriod=30
Jul 28 12:58:06.251303 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.251250    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kube-rbac-proxy" containerID="cri-o://f7457a37c113d9c295ec6206303e44de798480881a3f47126fc38c558890a7eb" gracePeriod=30
Jul 28 12:58:06.364873 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.364783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"]
Jul 28 12:58:06.368197 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.368181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.370800 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.370778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-da172-predictor-serving-cert\""
Jul 28 12:58:06.370912 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.370778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-da172-kube-rbac-proxy-sar-config\""
Jul 28 12:58:06.385146 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.385121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"]
Jul 28 12:58:06.408401 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.408375    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ef29225-a748-4be2-8e05-af974a483507" containerID="f7457a37c113d9c295ec6206303e44de798480881a3f47126fc38c558890a7eb" exitCode=2
Jul 28 12:58:06.408501 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.408413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" event={"ID":"9ef29225-a748-4be2-8e05-af974a483507","Type":"ContainerDied","Data":"f7457a37c113d9c295ec6206303e44de798480881a3f47126fc38c558890a7eb"}
Jul 28 12:58:06.496198 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.496169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4926a647-1726-415e-8835-5f65004ff82e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.496339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.496213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.496339 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.496245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7bc\" (UniqueName: \"kubernetes.io/projected/4926a647-1726-415e-8835-5f65004ff82e-kube-api-access-lc7bc\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.596705 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.596667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4926a647-1726-415e-8835-5f65004ff82e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.596896 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.596721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.596896 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.596758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7bc\" (UniqueName: \"kubernetes.io/projected/4926a647-1726-415e-8835-5f65004ff82e-kube-api-access-lc7bc\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.596995 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:06.596905    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-da172-predictor-serving-cert: secret "error-404-isvc-da172-predictor-serving-cert" not found
Jul 28 12:58:06.596995 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:06.596972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls podName:4926a647-1726-415e-8835-5f65004ff82e nodeName:}" failed. No retries permitted until 2026-07-28 12:58:07.096955302 +0000 UTC m=+1395.315307992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls") pod "error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" (UID: "4926a647-1726-415e-8835-5f65004ff82e") : secret "error-404-isvc-da172-predictor-serving-cert" not found
Jul 28 12:58:06.597311 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.597291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4926a647-1726-415e-8835-5f65004ff82e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:06.605521 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:06.605501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7bc\" (UniqueName: \"kubernetes.io/projected/4926a647-1726-415e-8835-5f65004ff82e-kube-api-access-lc7bc\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:07.101728 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:07.101681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:07.104260 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:07.104236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls\") pod \"error-404-isvc-da172-predictor-f8b85c5dd-tp4l2\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") " pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:07.283056 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:07.283018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:07.426897 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:07.426867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"]
Jul 28 12:58:07.430193 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:58:07.430167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4926a647_1726_415e_8835_5f65004ff82e.slice/crio-782623264cecf34d8e60fcde6576a1b42a23daa69d1f4eec995a4eed61cac043 WatchSource:0}: Error finding container 782623264cecf34d8e60fcde6576a1b42a23daa69d1f4eec995a4eed61cac043: Status 404 returned error can't find the container with id 782623264cecf34d8e60fcde6576a1b42a23daa69d1f4eec995a4eed61cac043
Jul 28 12:58:08.127138 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.127089    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:08.376202 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.376161    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:08.419668 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.419584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" event={"ID":"4926a647-1726-415e-8835-5f65004ff82e","Type":"ContainerStarted","Data":"fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e"}
Jul 28 12:58:08.419668 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.419648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" event={"ID":"4926a647-1726-415e-8835-5f65004ff82e","Type":"ContainerStarted","Data":"922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac"}
Jul 28 12:58:08.419923 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.419685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:08.419923 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.419711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" event={"ID":"4926a647-1726-415e-8835-5f65004ff82e","Type":"ContainerStarted","Data":"782623264cecf34d8e60fcde6576a1b42a23daa69d1f4eec995a4eed61cac043"}
Jul 28 12:58:08.419923 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.419853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:08.421134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.421108    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 12:58:08.441666 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:08.441613    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podStartSLOduration=2.441593925 podStartE2EDuration="2.441593925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:08.439887844 +0000 UTC m=+1396.658240556" watchObservedRunningTime="2026-07-28 12:58:08.441593925 +0000 UTC m=+1396.659946638"
Jul 28 12:58:09.427456 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.427428    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ef29225-a748-4be2-8e05-af974a483507" containerID="c2bb77740961941c08220939557ac2210895b5759840588e9939d91aaa936597" exitCode=0
Jul 28 12:58:09.427590 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.427516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" event={"ID":"9ef29225-a748-4be2-8e05-af974a483507","Type":"ContainerDied","Data":"c2bb77740961941c08220939557ac2210895b5759840588e9939d91aaa936597"}
Jul 28 12:58:09.428049 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.428024    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 12:58:09.513442 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.513387    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:58:09.624653 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.624619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls\") pod \"9ef29225-a748-4be2-8e05-af974a483507\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") "
Jul 28 12:58:09.624839 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.624663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qp7z\" (UniqueName: \"kubernetes.io/projected/9ef29225-a748-4be2-8e05-af974a483507-kube-api-access-9qp7z\") pod \"9ef29225-a748-4be2-8e05-af974a483507\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") "
Jul 28 12:58:09.624839 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.624691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ef29225-a748-4be2-8e05-af974a483507-kube-rbac-proxy-sar-config\") pod \"9ef29225-a748-4be2-8e05-af974a483507\" (UID: \"9ef29225-a748-4be2-8e05-af974a483507\") "
Jul 28 12:58:09.625309 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.625038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef29225-a748-4be2-8e05-af974a483507-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ef29225-a748-4be2-8e05-af974a483507" (UID: "9ef29225-a748-4be2-8e05-af974a483507"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:09.627026 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.627004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef29225-a748-4be2-8e05-af974a483507-kube-api-access-9qp7z" (OuterVolumeSpecName: "kube-api-access-9qp7z") pod "9ef29225-a748-4be2-8e05-af974a483507" (UID: "9ef29225-a748-4be2-8e05-af974a483507"). InnerVolumeSpecName "kube-api-access-9qp7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:58:09.627159 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.627139    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ef29225-a748-4be2-8e05-af974a483507" (UID: "9ef29225-a748-4be2-8e05-af974a483507"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:09.726140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.726109    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ef29225-a748-4be2-8e05-af974a483507-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:09.726140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.726136    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qp7z\" (UniqueName: \"kubernetes.io/projected/9ef29225-a748-4be2-8e05-af974a483507-kube-api-access-9qp7z\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:09.726140 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:09.726145    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ef29225-a748-4be2-8e05-af974a483507-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:10.433253 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:10.433222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"
Jul 28 12:58:10.433253 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:10.433247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r" event={"ID":"9ef29225-a748-4be2-8e05-af974a483507","Type":"ContainerDied","Data":"bd35fa893d2fecf9d851c861ca0dde6d576ce1b877456da4251d5ab258eae0ea"}
Jul 28 12:58:10.433492 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:10.433311    2563 scope.go:117] "RemoveContainer" containerID="f7457a37c113d9c295ec6206303e44de798480881a3f47126fc38c558890a7eb"
Jul 28 12:58:10.441751 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:10.441732    2563 scope.go:117] "RemoveContainer" containerID="c2bb77740961941c08220939557ac2210895b5759840588e9939d91aaa936597"
Jul 28 12:58:10.449725 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:10.449702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"]
Jul 28 12:58:10.453876 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:10.453854    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-75e21-predictor-64646f5b4b-6j67r"]
Jul 28 12:58:12.350716 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:12.350686    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef29225-a748-4be2-8e05-af974a483507" path="/var/lib/kubelet/pods/9ef29225-a748-4be2-8e05-af974a483507/volumes"
Jul 28 12:58:13.127236 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:13.127203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:13.376431 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:13.376394    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:14.432721 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:14.432695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:14.433207 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:14.433181    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 12:58:15.333386 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:15.333352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 12:58:16.275879 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.275856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:58:16.464798 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.464761    2563 generic.go:356] "Generic (PLEG): container finished" podID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerID="a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128" exitCode=0
Jul 28 12:58:16.465009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.464805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"
Jul 28 12:58:16.465009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.464810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" event={"ID":"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f","Type":"ContainerDied","Data":"a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128"}
Jul 28 12:58:16.465009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.464897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz" event={"ID":"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f","Type":"ContainerDied","Data":"d9e6dcf8892834962e9b1edc4d8d5a464cc6763ffc3cda774f186ff988b177e4"}
Jul 28 12:58:16.465009 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.464932    2563 scope.go:117] "RemoveContainer" containerID="a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128"
Jul 28 12:58:16.473494 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.473477    2563 scope.go:117] "RemoveContainer" containerID="a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128"
Jul 28 12:58:16.473759 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:16.473739    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128\": container with ID starting with a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128 not found: ID does not exist" containerID="a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128"
Jul 28 12:58:16.473880 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.473765    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128"} err="failed to get container status \"a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128\": rpc error: code = NotFound desc = could not find container \"a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128\": container with ID starting with a25cd1a40ef519712fbbfa4d9c66aa02e85c3121b05a52e48cee8a654e724128 not found: ID does not exist"
Jul 28 12:58:16.477022 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.477006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls\") pod \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") "
Jul 28 12:58:16.477086 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.477058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-openshift-service-ca-bundle\") pod \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\" (UID: \"6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f\") "
Jul 28 12:58:16.477374 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.477354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" (UID: "6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:16.479047 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.479029    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" (UID: "6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:16.578215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.578185    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:16.578215 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.578210    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:16.785313 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.785287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"]
Jul 28 12:58:16.791904 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:16.791882    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7fa9-649ffd79b6-dt6lz"]
Jul 28 12:58:18.344058 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:18.344023    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" path="/var/lib/kubelet/pods/6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f/volumes"
Jul 28 12:58:18.376206 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:18.376171    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:18.376321 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:18.376288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:58:23.375933 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:23.375894    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:24.433395 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:24.433362    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 12:58:25.333006 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:25.332957    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 12:58:28.376627 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:28.376562    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:33.376304 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:33.376254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:34.433892 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:34.433856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 12:58:35.334683 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:35.334651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:58:36.263164 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.263143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:58:36.343702 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.343675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls\") pod \"c92af120-de57-4b4d-8420-088863f54fe2\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") "
Jul 28 12:58:36.343902 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.343718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92af120-de57-4b4d-8420-088863f54fe2-openshift-service-ca-bundle\") pod \"c92af120-de57-4b4d-8420-088863f54fe2\" (UID: \"c92af120-de57-4b4d-8420-088863f54fe2\") "
Jul 28 12:58:36.344141 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.344110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92af120-de57-4b4d-8420-088863f54fe2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c92af120-de57-4b4d-8420-088863f54fe2" (UID: "c92af120-de57-4b4d-8420-088863f54fe2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:36.345997 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.345974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c92af120-de57-4b4d-8420-088863f54fe2" (UID: "c92af120-de57-4b4d-8420-088863f54fe2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:36.444605 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.444578    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c92af120-de57-4b4d-8420-088863f54fe2-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:36.444605 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.444605    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c92af120-de57-4b4d-8420-088863f54fe2-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:36.563815 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.563777    2563 generic.go:356] "Generic (PLEG): container finished" podID="c92af120-de57-4b4d-8420-088863f54fe2" containerID="d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a" exitCode=0
Jul 28 12:58:36.563815 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.563836    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"
Jul 28 12:58:36.564032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.563865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" event={"ID":"c92af120-de57-4b4d-8420-088863f54fe2","Type":"ContainerDied","Data":"d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a"}
Jul 28 12:58:36.564032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.563921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x" event={"ID":"c92af120-de57-4b4d-8420-088863f54fe2","Type":"ContainerDied","Data":"e99996034aa0e9a69c314f88aae1209282bbf518f9ad6d518b8b1ce3f1519916"}
Jul 28 12:58:36.564032 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.563944    2563 scope.go:117] "RemoveContainer" containerID="d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a"
Jul 28 12:58:36.572634 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.572618    2563 scope.go:117] "RemoveContainer" containerID="d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a"
Jul 28 12:58:36.572918 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:36.572900    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a\": container with ID starting with d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a not found: ID does not exist" containerID="d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a"
Jul 28 12:58:36.572998 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.572931    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a"} err="failed to get container status \"d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a\": rpc error: code = NotFound desc = could not find container \"d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a\": container with ID starting with d23e06c25abece0aec59e9afa3888808676f373359f4734e14385140202cad2a not found: ID does not exist"
Jul 28 12:58:36.587671 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.587642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"]
Jul 28 12:58:36.591593 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:36.591575    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-75e21-5bdd974b9c-4h42x"]
Jul 28 12:58:38.344271 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:38.344237    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c92af120-de57-4b4d-8420-088863f54fe2" path="/var/lib/kubelet/pods/c92af120-de57-4b4d-8420-088863f54fe2/volumes"
Jul 28 12:58:44.434016 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:44.433918    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 12:58:45.788953 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.788920    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"]
Jul 28 12:58:45.789450 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789430    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container"
Jul 28 12:58:45.789530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789453    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container"
Jul 28 12:58:45.789530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kube-rbac-proxy"
Jul 28 12:58:45.789530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789479    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kube-rbac-proxy"
Jul 28 12:58:45.789530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9"
Jul 28 12:58:45.789530 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9"
Jul 28 12:58:45.789772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789535    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21"
Jul 28 12:58:45.789772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21"
Jul 28 12:58:45.789772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789657    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6565bb02-bd4e-4b97-ab06-fbf3ce9b6f5f" containerName="sequence-graph-f7fa9"
Jul 28 12:58:45.789772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789673    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c92af120-de57-4b4d-8420-088863f54fe2" containerName="ensemble-graph-75e21"
Jul 28 12:58:45.789772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kserve-container"
Jul 28 12:58:45.789772 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.789692    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef29225-a748-4be2-8e05-af974a483507" containerName="kube-rbac-proxy"
Jul 28 12:58:45.794065 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.794035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:45.796098 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.796071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-fc1de-serving-cert\""
Jul 28 12:58:45.796209 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.796177    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-fc1de-kube-rbac-proxy-sar-config\""
Jul 28 12:58:45.800189 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.800167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"]
Jul 28 12:58:45.922793 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.922754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240bd96c-2163-4b91-bf1f-422c7c36e631-openshift-service-ca-bundle\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:45.922793 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:45.922797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.023979 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.023944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240bd96c-2163-4b91-bf1f-422c7c36e631-openshift-service-ca-bundle\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.024150 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.023988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.024194 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:46.024152    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-fc1de-serving-cert: secret "sequence-graph-fc1de-serving-cert" not found
Jul 28 12:58:46.024237 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:46.024227    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls podName:240bd96c-2163-4b91-bf1f-422c7c36e631 nodeName:}" failed. No retries permitted until 2026-07-28 12:58:46.524210203 +0000 UTC m=+1434.742562893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls") pod "sequence-graph-fc1de-b89b597bb-hvnqj" (UID: "240bd96c-2163-4b91-bf1f-422c7c36e631") : secret "sequence-graph-fc1de-serving-cert" not found
Jul 28 12:58:46.024556 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.024538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240bd96c-2163-4b91-bf1f-422c7c36e631-openshift-service-ca-bundle\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.527923 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.527887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.530516 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.530492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls\") pod \"sequence-graph-fc1de-b89b597bb-hvnqj\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") " pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.705536 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.705501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:46.837694 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:46.837624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"]
Jul 28 12:58:46.840008 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:58:46.839977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240bd96c_2163_4b91_bf1f_422c7c36e631.slice/crio-052520025b0e69164a1eda7de5f03bbc45fa534a53467f2729e50a4848eb3d22 WatchSource:0}: Error finding container 052520025b0e69164a1eda7de5f03bbc45fa534a53467f2729e50a4848eb3d22: Status 404 returned error can't find the container with id 052520025b0e69164a1eda7de5f03bbc45fa534a53467f2729e50a4848eb3d22
Jul 28 12:58:47.611907 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:47.611868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" event={"ID":"240bd96c-2163-4b91-bf1f-422c7c36e631","Type":"ContainerStarted","Data":"0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6"}
Jul 28 12:58:47.612082 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:47.611928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" event={"ID":"240bd96c-2163-4b91-bf1f-422c7c36e631","Type":"ContainerStarted","Data":"052520025b0e69164a1eda7de5f03bbc45fa534a53467f2729e50a4848eb3d22"}
Jul 28 12:58:47.612082 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:47.611976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:47.631841 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:47.631781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podStartSLOduration=2.631767904 podStartE2EDuration="2.631767904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:47.629639214 +0000 UTC m=+1435.847991971" watchObservedRunningTime="2026-07-28 12:58:47.631767904 +0000 UTC m=+1435.850120615"
Jul 28 12:58:53.622429 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:53.622402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:58:54.434997 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:54.434967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 12:58:55.871427 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:55.871398    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"]
Jul 28 12:58:55.871958 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:55.871593    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" containerID="cri-o://0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6" gracePeriod=30
Jul 28 12:58:56.051925 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.051890    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"]
Jul 28 12:58:56.052221 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.052194    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container" containerID="cri-o://4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933" gracePeriod=30
Jul 28 12:58:56.052338 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.052288    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kube-rbac-proxy" containerID="cri-o://a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758" gracePeriod=30
Jul 28 12:58:56.158033 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.157942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"]
Jul 28 12:58:56.162590 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.162568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.165017 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.164993    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d01ec-predictor-serving-cert\""
Jul 28 12:58:56.165186 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.165167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d01ec-kube-rbac-proxy-sar-config\""
Jul 28 12:58:56.171305 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.171282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"]
Jul 28 12:58:56.313635 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.313600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf6dd94-3518-4850-8caf-1cfab68c8671-proxy-tls\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.313814 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.313649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8sg4\" (UniqueName: \"kubernetes.io/projected/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-api-access-n8sg4\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.313814 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.313686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.414637 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.414546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf6dd94-3518-4850-8caf-1cfab68c8671-proxy-tls\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.414637 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.414606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8sg4\" (UniqueName: \"kubernetes.io/projected/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-api-access-n8sg4\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.414895 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.414666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.415320 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.415297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.416978 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.416949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf6dd94-3518-4850-8caf-1cfab68c8671-proxy-tls\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.424102 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.424076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8sg4\" (UniqueName: \"kubernetes.io/projected/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-api-access-n8sg4\") pod \"error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") " pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.472837 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.472790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:56.604050 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.604027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"]
Jul 28 12:58:56.606283 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:58:56.606261    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf6dd94_3518_4850_8caf_1cfab68c8671.slice/crio-729bb030abaeca6f125aac43e225df71ec9cda0d0fc412bbdeb279ace1e3c6f4 WatchSource:0}: Error finding container 729bb030abaeca6f125aac43e225df71ec9cda0d0fc412bbdeb279ace1e3c6f4: Status 404 returned error can't find the container with id 729bb030abaeca6f125aac43e225df71ec9cda0d0fc412bbdeb279ace1e3c6f4
Jul 28 12:58:56.652075 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.652047    2563 generic.go:356] "Generic (PLEG): container finished" podID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerID="a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758" exitCode=2
Jul 28 12:58:56.652199 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.652127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" event={"ID":"12056daa-b8ce-4537-b4dd-626a61607ecb","Type":"ContainerDied","Data":"a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758"}
Jul 28 12:58:56.653346 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:56.653328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" event={"ID":"fcf6dd94-3518-4850-8caf-1cfab68c8671","Type":"ContainerStarted","Data":"729bb030abaeca6f125aac43e225df71ec9cda0d0fc412bbdeb279ace1e3c6f4"}
Jul 28 12:58:57.659667 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:57.659633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" event={"ID":"fcf6dd94-3518-4850-8caf-1cfab68c8671","Type":"ContainerStarted","Data":"da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083"}
Jul 28 12:58:57.660061 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:57.659690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" event={"ID":"fcf6dd94-3518-4850-8caf-1cfab68c8671","Type":"ContainerStarted","Data":"4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e"}
Jul 28 12:58:57.660061 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:57.659714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:57.679423 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:57.679373    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podStartSLOduration=1.679358974 podStartE2EDuration="1.679358974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:57.678126675 +0000 UTC m=+1445.896479388" watchObservedRunningTime="2026-07-28 12:58:57.679358974 +0000 UTC m=+1445.897711685"
Jul 28 12:58:58.620728 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:58.620691    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:58:58.664774 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:58.664748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:58:58.666248 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:58.666221    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 12:58:59.404078 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.404049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:58:59.541485 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.541456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-rbac-proxy-sar-config\") pod \"12056daa-b8ce-4537-b4dd-626a61607ecb\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") "
Jul 28 12:58:59.541683 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.541492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") pod \"12056daa-b8ce-4537-b4dd-626a61607ecb\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") "
Jul 28 12:58:59.541683 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.541539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htp55\" (UniqueName: \"kubernetes.io/projected/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-api-access-htp55\") pod \"12056daa-b8ce-4537-b4dd-626a61607ecb\" (UID: \"12056daa-b8ce-4537-b4dd-626a61607ecb\") "
Jul 28 12:58:59.541804 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.541784    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12056daa-b8ce-4537-b4dd-626a61607ecb" (UID: "12056daa-b8ce-4537-b4dd-626a61607ecb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:59.543841 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.543804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12056daa-b8ce-4537-b4dd-626a61607ecb" (UID: "12056daa-b8ce-4537-b4dd-626a61607ecb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:59.543957 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.543839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-api-access-htp55" (OuterVolumeSpecName: "kube-api-access-htp55") pod "12056daa-b8ce-4537-b4dd-626a61607ecb" (UID: "12056daa-b8ce-4537-b4dd-626a61607ecb"). InnerVolumeSpecName "kube-api-access-htp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:58:59.642748 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.642718    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:59.642748 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.642743    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12056daa-b8ce-4537-b4dd-626a61607ecb-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:59.642748 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.642755    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htp55\" (UniqueName: \"kubernetes.io/projected/12056daa-b8ce-4537-b4dd-626a61607ecb-kube-api-access-htp55\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:58:59.670704 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.670672    2563 generic.go:356] "Generic (PLEG): container finished" podID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerID="4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933" exitCode=0
Jul 28 12:58:59.671134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.670740    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"
Jul 28 12:58:59.671134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.670758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" event={"ID":"12056daa-b8ce-4537-b4dd-626a61607ecb","Type":"ContainerDied","Data":"4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933"}
Jul 28 12:58:59.671134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.670816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk" event={"ID":"12056daa-b8ce-4537-b4dd-626a61607ecb","Type":"ContainerDied","Data":"22c7128da742fd2a341d9beca18673469ec0ec70db23665195910647f9ef9d3a"}
Jul 28 12:58:59.671134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.670863    2563 scope.go:117] "RemoveContainer" containerID="a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758"
Jul 28 12:58:59.671513 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.671478    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 12:58:59.683366 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.683350    2563 scope.go:117] "RemoveContainer" containerID="4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933"
Jul 28 12:58:59.690585 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.690569    2563 scope.go:117] "RemoveContainer" containerID="a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758"
Jul 28 12:58:59.690870 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:59.690850    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758\": container with ID starting with a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758 not found: ID does not exist" containerID="a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758"
Jul 28 12:58:59.690948 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.690878    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758"} err="failed to get container status \"a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758\": rpc error: code = NotFound desc = could not find container \"a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758\": container with ID starting with a87ea23558519310ad0e8845ab1633402ccfa21681f520f84efe321559a33758 not found: ID does not exist"
Jul 28 12:58:59.690948 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.690895    2563 scope.go:117] "RemoveContainer" containerID="4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933"
Jul 28 12:58:59.691130 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:58:59.691115    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933\": container with ID starting with 4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933 not found: ID does not exist" containerID="4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933"
Jul 28 12:58:59.691168 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.691135    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933"} err="failed to get container status \"4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933\": rpc error: code = NotFound desc = could not find container \"4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933\": container with ID starting with 4f8bfd25e5e93c4e2d7dda5be27b6ef86fbbdd4f3dbcb25e27ce50c89df1d933 not found: ID does not exist"
Jul 28 12:58:59.695844 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.695811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"]
Jul 28 12:58:59.706938 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:58:59.706920    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc1de-predictor-6bc584d948-4fgbk"]
Jul 28 12:59:00.344168 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:00.344137    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" path="/var/lib/kubelet/pods/12056daa-b8ce-4537-b4dd-626a61607ecb/volumes"
Jul 28 12:59:03.620504 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:03.620460    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:59:04.677279 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:04.677253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:59:04.677761 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:04.677727    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 12:59:06.331749 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.331712    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"]
Jul 28 12:59:06.333088 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.333063    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container"
Jul 28 12:59:06.333241 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.333230    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container"
Jul 28 12:59:06.333326 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.333317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kube-rbac-proxy"
Jul 28 12:59:06.333396 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.333388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kube-rbac-proxy"
Jul 28 12:59:06.333713 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.333699    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kserve-container"
Jul 28 12:59:06.333809 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.333800    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12056daa-b8ce-4537-b4dd-626a61607ecb" containerName="kube-rbac-proxy"
Jul 28 12:59:06.339434 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.339404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.341773 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.341751    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-da172-serving-cert\""
Jul 28 12:59:06.341908 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.341778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-da172-kube-rbac-proxy-sar-config\""
Jul 28 12:59:06.347154 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.347127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"]
Jul 28 12:59:06.501466 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.501433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c019bab-57fc-45cf-bc44-48d81238c31c-openshift-service-ca-bundle\") pod \"ensemble-graph-da172-54c5896848-8vfjc\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") " pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.501648 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.501478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c019bab-57fc-45cf-bc44-48d81238c31c-proxy-tls\") pod \"ensemble-graph-da172-54c5896848-8vfjc\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") " pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.602480 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.602374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c019bab-57fc-45cf-bc44-48d81238c31c-openshift-service-ca-bundle\") pod \"ensemble-graph-da172-54c5896848-8vfjc\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") " pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.602480 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.602431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c019bab-57fc-45cf-bc44-48d81238c31c-proxy-tls\") pod \"ensemble-graph-da172-54c5896848-8vfjc\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") " pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.603157 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.603131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c019bab-57fc-45cf-bc44-48d81238c31c-openshift-service-ca-bundle\") pod \"ensemble-graph-da172-54c5896848-8vfjc\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") " pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.605044 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.605025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c019bab-57fc-45cf-bc44-48d81238c31c-proxy-tls\") pod \"ensemble-graph-da172-54c5896848-8vfjc\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") " pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.651282 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.651249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:06.787368 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:06.787178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"]
Jul 28 12:59:06.790071 ip-10-0-137-92 kubenswrapper[2563]: W0728 12:59:06.790034    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c019bab_57fc_45cf_bc44_48d81238c31c.slice/crio-70dd0519a25ac4e30a2b1c53cbc6f215c5a7978de1b62ed39ceb01bc2d079a84 WatchSource:0}: Error finding container 70dd0519a25ac4e30a2b1c53cbc6f215c5a7978de1b62ed39ceb01bc2d079a84: Status 404 returned error can't find the container with id 70dd0519a25ac4e30a2b1c53cbc6f215c5a7978de1b62ed39ceb01bc2d079a84
Jul 28 12:59:07.712134 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:07.712099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" event={"ID":"2c019bab-57fc-45cf-bc44-48d81238c31c","Type":"ContainerStarted","Data":"809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4"}
Jul 28 12:59:07.712531 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:07.712149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" event={"ID":"2c019bab-57fc-45cf-bc44-48d81238c31c","Type":"ContainerStarted","Data":"70dd0519a25ac4e30a2b1c53cbc6f215c5a7978de1b62ed39ceb01bc2d079a84"}
Jul 28 12:59:07.712531 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:07.712184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:07.730939 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:07.730899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podStartSLOduration=1.730884794 podStartE2EDuration="1.730884794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:59:07.730213747 +0000 UTC m=+1455.948566460" watchObservedRunningTime="2026-07-28 12:59:07.730884794 +0000 UTC m=+1455.949237505"
Jul 28 12:59:08.620295 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:08.620256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:59:08.620488 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:08.620364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:59:13.620680 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:13.620632    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:59:13.721977 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:13.721943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 12:59:14.678557 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:14.678517    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 12:59:18.620496 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:18.620450    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:59:23.620365 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:23.620324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:59:24.677893 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:24.677856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 12:59:26.014940 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.014914    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:59:26.059492 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.059461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240bd96c-2163-4b91-bf1f-422c7c36e631-openshift-service-ca-bundle\") pod \"240bd96c-2163-4b91-bf1f-422c7c36e631\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") "
Jul 28 12:59:26.059671 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.059549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls\") pod \"240bd96c-2163-4b91-bf1f-422c7c36e631\" (UID: \"240bd96c-2163-4b91-bf1f-422c7c36e631\") "
Jul 28 12:59:26.059882 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.059837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240bd96c-2163-4b91-bf1f-422c7c36e631-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "240bd96c-2163-4b91-bf1f-422c7c36e631" (UID: "240bd96c-2163-4b91-bf1f-422c7c36e631"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:59:26.061853 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.061811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "240bd96c-2163-4b91-bf1f-422c7c36e631" (UID: "240bd96c-2163-4b91-bf1f-422c7c36e631"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:59:26.160267 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.160190    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240bd96c-2163-4b91-bf1f-422c7c36e631-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:59:26.160267 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.160214    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/240bd96c-2163-4b91-bf1f-422c7c36e631-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 12:59:26.798273 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.798239    2563 generic.go:356] "Generic (PLEG): container finished" podID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerID="0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6" exitCode=0
Jul 28 12:59:26.798525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.798284    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"
Jul 28 12:59:26.798525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.798325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" event={"ID":"240bd96c-2163-4b91-bf1f-422c7c36e631","Type":"ContainerDied","Data":"0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6"}
Jul 28 12:59:26.798525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.798377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj" event={"ID":"240bd96c-2163-4b91-bf1f-422c7c36e631","Type":"ContainerDied","Data":"052520025b0e69164a1eda7de5f03bbc45fa534a53467f2729e50a4848eb3d22"}
Jul 28 12:59:26.798525 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.798399    2563 scope.go:117] "RemoveContainer" containerID="0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6"
Jul 28 12:59:26.806754 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.806736    2563 scope.go:117] "RemoveContainer" containerID="0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6"
Jul 28 12:59:26.807053 ip-10-0-137-92 kubenswrapper[2563]: E0728 12:59:26.807037    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6\": container with ID starting with 0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6 not found: ID does not exist" containerID="0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6"
Jul 28 12:59:26.807124 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.807061    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6"} err="failed to get container status \"0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6\": rpc error: code = NotFound desc = could not find container \"0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6\": container with ID starting with 0f0791fee2b733a97ad247583d4b519a005095108e47c71d088bca16080b21d6 not found: ID does not exist"
Jul 28 12:59:26.817220 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.817199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"]
Jul 28 12:59:26.823536 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:26.823510    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fc1de-b89b597bb-hvnqj"]
Jul 28 12:59:28.344564 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:28.344533    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" path="/var/lib/kubelet/pods/240bd96c-2163-4b91-bf1f-422c7c36e631/volumes"
Jul 28 12:59:34.678672 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:34.678625    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 12:59:44.679552 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:44.679523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 12:59:52.335752 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:52.335723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 12:59:52.338734 ip-10-0-137-92 kubenswrapper[2563]: I0728 12:59:52.338713    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:00:06.140509 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.140431    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"]
Jul 28 13:00:06.140869 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.140770    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de"
Jul 28 13:00:06.140869 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.140780    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de"
Jul 28 13:00:06.140951 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.140876    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="240bd96c-2163-4b91-bf1f-422c7c36e631" containerName="sequence-graph-fc1de"
Jul 28 13:00:06.143915 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.143899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.146325 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.146291    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d01ec-serving-cert\""
Jul 28 13:00:06.146442 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.146330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d01ec-kube-rbac-proxy-sar-config\""
Jul 28 13:00:06.155177 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.155151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"]
Jul 28 13:00:06.205134 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.205104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1878b56-1edb-4f07-af9c-b7c807c95532-openshift-service-ca-bundle\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.205266 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.205146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.306250 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.306216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1878b56-1edb-4f07-af9c-b7c807c95532-openshift-service-ca-bundle\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.306408 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.306264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.306450 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:00:06.306405    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-d01ec-serving-cert: secret "sequence-graph-d01ec-serving-cert" not found
Jul 28 13:00:06.306490 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:00:06.306461    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls podName:c1878b56-1edb-4f07-af9c-b7c807c95532 nodeName:}" failed. No retries permitted until 2026-07-28 13:00:06.806444703 +0000 UTC m=+1515.024797392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls") pod "sequence-graph-d01ec-59754655f9-2xm8b" (UID: "c1878b56-1edb-4f07-af9c-b7c807c95532") : secret "sequence-graph-d01ec-serving-cert" not found
Jul 28 13:00:06.306911 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.306893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1878b56-1edb-4f07-af9c-b7c807c95532-openshift-service-ca-bundle\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.810895 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.810863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:06.813431 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:06.813396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls\") pod \"sequence-graph-d01ec-59754655f9-2xm8b\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") " pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:07.055205 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:07.055180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:07.186757 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:07.186733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"]
Jul 28 13:00:07.189019 ip-10-0-137-92 kubenswrapper[2563]: W0728 13:00:07.188986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1878b56_1edb_4f07_af9c_b7c807c95532.slice/crio-f328141c5b93d57775788ed4e5c70e2dd8460df75902a7d17579b2243d1ccc45 WatchSource:0}: Error finding container f328141c5b93d57775788ed4e5c70e2dd8460df75902a7d17579b2243d1ccc45: Status 404 returned error can't find the container with id f328141c5b93d57775788ed4e5c70e2dd8460df75902a7d17579b2243d1ccc45
Jul 28 13:00:07.974597 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:07.974564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" event={"ID":"c1878b56-1edb-4f07-af9c-b7c807c95532","Type":"ContainerStarted","Data":"6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19"}
Jul 28 13:00:07.974810 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:07.974652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" event={"ID":"c1878b56-1edb-4f07-af9c-b7c807c95532","Type":"ContainerStarted","Data":"f328141c5b93d57775788ed4e5c70e2dd8460df75902a7d17579b2243d1ccc45"}
Jul 28 13:00:07.974810 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:07.974695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:00:08.001789 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:08.001738    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podStartSLOduration=2.001723618 podStartE2EDuration="2.001723618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:08.000141596 +0000 UTC m=+1516.218494337" watchObservedRunningTime="2026-07-28 13:00:08.001723618 +0000 UTC m=+1516.220076331"
Jul 28 13:00:13.986156 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:00:13.986072    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:04:52.363207 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:04:52.363182    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:04:52.367062 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:04:52.367042    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:07:21.003362 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.003329    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"]
Jul 28 13:07:21.003964 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.003530    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" containerID="cri-o://809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4" gracePeriod=30
Jul 28 13:07:21.249850 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.249806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"]
Jul 28 13:07:21.253293 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.253275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.257313 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.257257    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4b3a0-kube-rbac-proxy-sar-config\""
Jul 28 13:07:21.257313 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.257292    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4b3a0-predictor-serving-cert\""
Jul 28 13:07:21.263190 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.263168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"]
Jul 28 13:07:21.294896 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.294868    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"]
Jul 28 13:07:21.295113 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.295071    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" containerID="cri-o://922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac" gracePeriod=30
Jul 28 13:07:21.295216 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.295112    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kube-rbac-proxy" containerID="cri-o://fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e" gracePeriod=30
Jul 28 13:07:21.385794 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.385760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b966603-3307-401a-ae83-d8ae74c7c7b1-proxy-tls\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.385794 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.385807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvn8\" (UniqueName: \"kubernetes.io/projected/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-api-access-5cvn8\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.386029 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.385866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.486940 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.486905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.487087 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.486991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b966603-3307-401a-ae83-d8ae74c7c7b1-proxy-tls\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.487087 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.487021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvn8\" (UniqueName: \"kubernetes.io/projected/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-api-access-5cvn8\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.487527 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.487497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.489589 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.489571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b966603-3307-401a-ae83-d8ae74c7c7b1-proxy-tls\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.496392 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.496365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvn8\" (UniqueName: \"kubernetes.io/projected/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-api-access-5cvn8\") pod \"error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") " pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.564125 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.564048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.695948 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.695920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"]
Jul 28 13:07:21.697862 ip-10-0-137-92 kubenswrapper[2563]: W0728 13:07:21.697811    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b966603_3307_401a_ae83_d8ae74c7c7b1.slice/crio-3e8f032d1280079e26868b4eec423fa1303f9c246752f0f7c9faf081ace650d5 WatchSource:0}: Error finding container 3e8f032d1280079e26868b4eec423fa1303f9c246752f0f7c9faf081ace650d5: Status 404 returned error can't find the container with id 3e8f032d1280079e26868b4eec423fa1303f9c246752f0f7c9faf081ace650d5
Jul 28 13:07:21.869980 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.869943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" event={"ID":"9b966603-3307-401a-ae83-d8ae74c7c7b1","Type":"ContainerStarted","Data":"a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac"}
Jul 28 13:07:21.870157 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.870024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" event={"ID":"9b966603-3307-401a-ae83-d8ae74c7c7b1","Type":"ContainerStarted","Data":"4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6"}
Jul 28 13:07:21.870157 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.870055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" event={"ID":"9b966603-3307-401a-ae83-d8ae74c7c7b1","Type":"ContainerStarted","Data":"3e8f032d1280079e26868b4eec423fa1303f9c246752f0f7c9faf081ace650d5"}
Jul 28 13:07:21.870157 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.870085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.870157 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.870122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:21.872109 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.872075    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:07:21.872263 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.872242    2563 generic.go:356] "Generic (PLEG): container finished" podID="4926a647-1726-415e-8835-5f65004ff82e" containerID="fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e" exitCode=2
Jul 28 13:07:21.872328 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.872307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" event={"ID":"4926a647-1726-415e-8835-5f65004ff82e","Type":"ContainerDied","Data":"fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e"}
Jul 28 13:07:21.887919 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:21.887873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podStartSLOduration=0.887857876 podStartE2EDuration="887.857876ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:07:21.886621966 +0000 UTC m=+1950.104974677" watchObservedRunningTime="2026-07-28 13:07:21.887857876 +0000 UTC m=+1950.106210588"
Jul 28 13:07:22.876715 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:22.876671    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:07:23.720615 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:23.720575    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:24.453002 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.452978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 13:07:24.509599 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.509568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4926a647-1726-415e-8835-5f65004ff82e-kube-rbac-proxy-sar-config\") pod \"4926a647-1726-415e-8835-5f65004ff82e\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") "
Jul 28 13:07:24.509746 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.509617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc7bc\" (UniqueName: \"kubernetes.io/projected/4926a647-1726-415e-8835-5f65004ff82e-kube-api-access-lc7bc\") pod \"4926a647-1726-415e-8835-5f65004ff82e\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") "
Jul 28 13:07:24.509746 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.509655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls\") pod \"4926a647-1726-415e-8835-5f65004ff82e\" (UID: \"4926a647-1726-415e-8835-5f65004ff82e\") "
Jul 28 13:07:24.509979 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.509955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4926a647-1726-415e-8835-5f65004ff82e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4926a647-1726-415e-8835-5f65004ff82e" (UID: "4926a647-1726-415e-8835-5f65004ff82e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:24.511757 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.511721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4926a647-1726-415e-8835-5f65004ff82e-kube-api-access-lc7bc" (OuterVolumeSpecName: "kube-api-access-lc7bc") pod "4926a647-1726-415e-8835-5f65004ff82e" (UID: "4926a647-1726-415e-8835-5f65004ff82e"). InnerVolumeSpecName "kube-api-access-lc7bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:07:24.511868 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.511809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4926a647-1726-415e-8835-5f65004ff82e" (UID: "4926a647-1726-415e-8835-5f65004ff82e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:07:24.610929 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.610902    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4926a647-1726-415e-8835-5f65004ff82e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:07:24.610929 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.610924    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7bc\" (UniqueName: \"kubernetes.io/projected/4926a647-1726-415e-8835-5f65004ff82e-kube-api-access-lc7bc\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:07:24.610929 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.610934    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4926a647-1726-415e-8835-5f65004ff82e-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:07:24.887343 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.887257    2563 generic.go:356] "Generic (PLEG): container finished" podID="4926a647-1726-415e-8835-5f65004ff82e" containerID="922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac" exitCode=0
Jul 28 13:07:24.887343 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.887310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"
Jul 28 13:07:24.887565 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.887313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" event={"ID":"4926a647-1726-415e-8835-5f65004ff82e","Type":"ContainerDied","Data":"922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac"}
Jul 28 13:07:24.887565 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.887461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" event={"ID":"4926a647-1726-415e-8835-5f65004ff82e","Type":"ContainerDied","Data":"782623264cecf34d8e60fcde6576a1b42a23daa69d1f4eec995a4eed61cac043"}
Jul 28 13:07:24.887565 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.887504    2563 scope.go:117] "RemoveContainer" containerID="fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e"
Jul 28 13:07:24.896906 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.896861    2563 scope.go:117] "RemoveContainer" containerID="922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac"
Jul 28 13:07:24.904288 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.904273    2563 scope.go:117] "RemoveContainer" containerID="fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e"
Jul 28 13:07:24.904547 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:07:24.904524    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e\": container with ID starting with fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e not found: ID does not exist" containerID="fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e"
Jul 28 13:07:24.904630 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.904568    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e"} err="failed to get container status \"fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e\": rpc error: code = NotFound desc = could not find container \"fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e\": container with ID starting with fcbc6551a508e0b54c70285dfe7dcee8b64ff4adacba1e1c3e05de6a7d968b1e not found: ID does not exist"
Jul 28 13:07:24.904630 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.904595    2563 scope.go:117] "RemoveContainer" containerID="922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac"
Jul 28 13:07:24.904875 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:07:24.904857    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac\": container with ID starting with 922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac not found: ID does not exist" containerID="922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac"
Jul 28 13:07:24.904944 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.904883    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac"} err="failed to get container status \"922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac\": rpc error: code = NotFound desc = could not find container \"922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac\": container with ID starting with 922d4a09c8df1b3673040790a3e1bbf212fb487936ca3db5f43da90e0bf46dac not found: ID does not exist"
Jul 28 13:07:24.908147 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.908127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"]
Jul 28 13:07:24.912224 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:24.912205    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2"]
Jul 28 13:07:25.428950 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:25.428907    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": context deadline exceeded"
Jul 28 13:07:25.434205 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:25.434181    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da172-predictor-f8b85c5dd-tp4l2" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: i/o timeout"
Jul 28 13:07:26.344518 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:26.344490    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4926a647-1726-415e-8835-5f65004ff82e" path="/var/lib/kubelet/pods/4926a647-1726-415e-8835-5f65004ff82e/volumes"
Jul 28 13:07:27.880924 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:27.880900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:07:27.881354 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:27.881320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:07:28.721755 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:28.721714    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:33.721406 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:33.721365    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:33.721806 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:33.721478    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 13:07:37.881908 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:37.881866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:07:38.721424 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:38.721382    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:43.721540 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:43.721450    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:47.881732 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:47.881695    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:07:48.721038 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:48.721001    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:07:51.146989 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.146965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 13:07:51.235957 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.235924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c019bab-57fc-45cf-bc44-48d81238c31c-proxy-tls\") pod \"2c019bab-57fc-45cf-bc44-48d81238c31c\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") "
Jul 28 13:07:51.236141 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.235982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c019bab-57fc-45cf-bc44-48d81238c31c-openshift-service-ca-bundle\") pod \"2c019bab-57fc-45cf-bc44-48d81238c31c\" (UID: \"2c019bab-57fc-45cf-bc44-48d81238c31c\") "
Jul 28 13:07:51.236346 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.236319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c019bab-57fc-45cf-bc44-48d81238c31c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2c019bab-57fc-45cf-bc44-48d81238c31c" (UID: "2c019bab-57fc-45cf-bc44-48d81238c31c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:51.238145 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.238116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c019bab-57fc-45cf-bc44-48d81238c31c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c019bab-57fc-45cf-bc44-48d81238c31c" (UID: "2c019bab-57fc-45cf-bc44-48d81238c31c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:07:51.336884 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.336798    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c019bab-57fc-45cf-bc44-48d81238c31c-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:07:51.336884 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:51.336846    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c019bab-57fc-45cf-bc44-48d81238c31c-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:07:52.015771 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.015738    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerID="809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4" exitCode=0
Jul 28 13:07:52.016061 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.015807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"
Jul 28 13:07:52.016061 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.015848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" event={"ID":"2c019bab-57fc-45cf-bc44-48d81238c31c","Type":"ContainerDied","Data":"809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4"}
Jul 28 13:07:52.016061 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.015904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc" event={"ID":"2c019bab-57fc-45cf-bc44-48d81238c31c","Type":"ContainerDied","Data":"70dd0519a25ac4e30a2b1c53cbc6f215c5a7978de1b62ed39ceb01bc2d079a84"}
Jul 28 13:07:52.016061 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.015926    2563 scope.go:117] "RemoveContainer" containerID="809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4"
Jul 28 13:07:52.024967 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.024951    2563 scope.go:117] "RemoveContainer" containerID="809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4"
Jul 28 13:07:52.025211 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:07:52.025193    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4\": container with ID starting with 809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4 not found: ID does not exist" containerID="809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4"
Jul 28 13:07:52.025275 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.025219    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4"} err="failed to get container status \"809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4\": rpc error: code = NotFound desc = could not find container \"809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4\": container with ID starting with 809462d9d99ce908edab24cff717523a6a229dce7254920eeda6840eead70dc4 not found: ID does not exist"
Jul 28 13:07:52.037313 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.037291    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"]
Jul 28 13:07:52.040962 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.040943    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-da172-54c5896848-8vfjc"]
Jul 28 13:07:52.345382 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:52.345308    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" path="/var/lib/kubelet/pods/2c019bab-57fc-45cf-bc44-48d81238c31c/volumes"
Jul 28 13:07:57.881768 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:07:57.881723    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 13:08:07.882440 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:07.882412    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:08:20.714998 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:20.714967    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"]
Jul 28 13:08:20.715397 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:20.715178    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" containerID="cri-o://6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19" gracePeriod=30
Jul 28 13:08:20.791092 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:20.791059    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"]
Jul 28 13:08:20.791329 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:20.791304    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container" containerID="cri-o://4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e" gracePeriod=30
Jul 28 13:08:20.791435 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:20.791349    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kube-rbac-proxy" containerID="cri-o://da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083" gracePeriod=30
Jul 28 13:08:21.021007 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.020919    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"]
Jul 28 13:08:21.021362 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kube-rbac-proxy"
Jul 28 13:08:21.021448 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kube-rbac-proxy"
Jul 28 13:08:21.021448 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container"
Jul 28 13:08:21.021448 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021391    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container"
Jul 28 13:08:21.021448 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021423    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172"
Jul 28 13:08:21.021448 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021431    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172"
Jul 28 13:08:21.021698 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021520    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kserve-container"
Jul 28 13:08:21.021698 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021533    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c019bab-57fc-45cf-bc44-48d81238c31c" containerName="ensemble-graph-da172"
Jul 28 13:08:21.021698 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.021544    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4926a647-1726-415e-8835-5f65004ff82e" containerName="kube-rbac-proxy"
Jul 28 13:08:21.024557 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.024534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.026839 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.026806    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dde97-kube-rbac-proxy-sar-config\""
Jul 28 13:08:21.026923 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.026902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dde97-predictor-serving-cert\""
Jul 28 13:08:21.039422 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.039392    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"]
Jul 28 13:08:21.080602 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.080570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507c16a1-485a-40ae-a166-4a0a8802217f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.080767 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.080694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507c16a1-485a-40ae-a166-4a0a8802217f-proxy-tls\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.080767 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.080719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhb8g\" (UniqueName: \"kubernetes.io/projected/507c16a1-485a-40ae-a166-4a0a8802217f-kube-api-access-dhb8g\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.152989 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.152954    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerID="da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083" exitCode=2
Jul 28 13:08:21.153150 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.153000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" event={"ID":"fcf6dd94-3518-4850-8caf-1cfab68c8671","Type":"ContainerDied","Data":"da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083"}
Jul 28 13:08:21.182204 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.182166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507c16a1-485a-40ae-a166-4a0a8802217f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.182379 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.182310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507c16a1-485a-40ae-a166-4a0a8802217f-proxy-tls\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.182379 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.182342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhb8g\" (UniqueName: \"kubernetes.io/projected/507c16a1-485a-40ae-a166-4a0a8802217f-kube-api-access-dhb8g\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.182949 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.182922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507c16a1-485a-40ae-a166-4a0a8802217f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.185157 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.185124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507c16a1-485a-40ae-a166-4a0a8802217f-proxy-tls\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.193611 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.193592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhb8g\" (UniqueName: \"kubernetes.io/projected/507c16a1-485a-40ae-a166-4a0a8802217f-kube-api-access-dhb8g\") pod \"error-404-isvc-dde97-predictor-db9fd4f9b-7kx85\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") " pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.341710 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.341633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:21.476659 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:21.476631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"]
Jul 28 13:08:21.478908 ip-10-0-137-92 kubenswrapper[2563]: W0728 13:08:21.478885    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507c16a1_485a_40ae_a166_4a0a8802217f.slice/crio-10817aa8f442558639674b8009f9df462386c7ecf160028f6e2971cd02f25d59 WatchSource:0}: Error finding container 10817aa8f442558639674b8009f9df462386c7ecf160028f6e2971cd02f25d59: Status 404 returned error can't find the container with id 10817aa8f442558639674b8009f9df462386c7ecf160028f6e2971cd02f25d59
Jul 28 13:08:22.159456 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:22.159420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" event={"ID":"507c16a1-485a-40ae-a166-4a0a8802217f","Type":"ContainerStarted","Data":"4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7"}
Jul 28 13:08:22.159908 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:22.159475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" event={"ID":"507c16a1-485a-40ae-a166-4a0a8802217f","Type":"ContainerStarted","Data":"9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4"}
Jul 28 13:08:22.159908 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:22.159494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" event={"ID":"507c16a1-485a-40ae-a166-4a0a8802217f","Type":"ContainerStarted","Data":"10817aa8f442558639674b8009f9df462386c7ecf160028f6e2971cd02f25d59"}
Jul 28 13:08:22.159908 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:22.159555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:22.176537 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:22.176490    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podStartSLOduration=1.176476944 podStartE2EDuration="1.176476944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:22.175467569 +0000 UTC m=+2010.393820281" watchObservedRunningTime="2026-07-28 13:08:22.176476944 +0000 UTC m=+2010.394829655"
Jul 28 13:08:23.165168 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:23.165137    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:23.166537 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:23.166503    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:08:23.936475 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:23.936449    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 13:08:23.984035 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:23.984000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:24.005983 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.005957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-rbac-proxy-sar-config\") pod \"fcf6dd94-3518-4850-8caf-1cfab68c8671\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") "
Jul 28 13:08:24.006112 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.006002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf6dd94-3518-4850-8caf-1cfab68c8671-proxy-tls\") pod \"fcf6dd94-3518-4850-8caf-1cfab68c8671\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") "
Jul 28 13:08:24.006112 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.006039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8sg4\" (UniqueName: \"kubernetes.io/projected/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-api-access-n8sg4\") pod \"fcf6dd94-3518-4850-8caf-1cfab68c8671\" (UID: \"fcf6dd94-3518-4850-8caf-1cfab68c8671\") "
Jul 28 13:08:24.006388 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.006360    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcf6dd94-3518-4850-8caf-1cfab68c8671" (UID: "fcf6dd94-3518-4850-8caf-1cfab68c8671"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:24.008303 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.008279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-api-access-n8sg4" (OuterVolumeSpecName: "kube-api-access-n8sg4") pod "fcf6dd94-3518-4850-8caf-1cfab68c8671" (UID: "fcf6dd94-3518-4850-8caf-1cfab68c8671"). InnerVolumeSpecName "kube-api-access-n8sg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:24.008396 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.008376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf6dd94-3518-4850-8caf-1cfab68c8671-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcf6dd94-3518-4850-8caf-1cfab68c8671" (UID: "fcf6dd94-3518-4850-8caf-1cfab68c8671"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:24.106581 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.106510    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:24.106581 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.106535    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf6dd94-3518-4850-8caf-1cfab68c8671-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:24.106581 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.106545    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8sg4\" (UniqueName: \"kubernetes.io/projected/fcf6dd94-3518-4850-8caf-1cfab68c8671-kube-api-access-n8sg4\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:24.171188 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.171147    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerID="4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e" exitCode=0
Jul 28 13:08:24.171572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.171218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"
Jul 28 13:08:24.171572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.171236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" event={"ID":"fcf6dd94-3518-4850-8caf-1cfab68c8671","Type":"ContainerDied","Data":"4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e"}
Jul 28 13:08:24.171572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.171297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2" event={"ID":"fcf6dd94-3518-4850-8caf-1cfab68c8671","Type":"ContainerDied","Data":"729bb030abaeca6f125aac43e225df71ec9cda0d0fc412bbdeb279ace1e3c6f4"}
Jul 28 13:08:24.171572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.171335    2563 scope.go:117] "RemoveContainer" containerID="da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083"
Jul 28 13:08:24.171837 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.171767    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:08:24.180412 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.180399    2563 scope.go:117] "RemoveContainer" containerID="4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e"
Jul 28 13:08:24.188072 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.188056    2563 scope.go:117] "RemoveContainer" containerID="da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083"
Jul 28 13:08:24.188303 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:08:24.188284    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083\": container with ID starting with da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083 not found: ID does not exist" containerID="da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083"
Jul 28 13:08:24.188345 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.188312    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083"} err="failed to get container status \"da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083\": rpc error: code = NotFound desc = could not find container \"da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083\": container with ID starting with da3e734df76f7e22d289c2bb17810224942dd853f4aa5fcd9516f7a76c5f4083 not found: ID does not exist"
Jul 28 13:08:24.188345 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.188329    2563 scope.go:117] "RemoveContainer" containerID="4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e"
Jul 28 13:08:24.188570 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:08:24.188554    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e\": container with ID starting with 4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e not found: ID does not exist" containerID="4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e"
Jul 28 13:08:24.188610 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.188579    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e"} err="failed to get container status \"4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e\": rpc error: code = NotFound desc = could not find container \"4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e\": container with ID starting with 4fd2e9a11920864bd342e838ed62fae8106d650a20175bd7fff585827d6a8b4e not found: ID does not exist"
Jul 28 13:08:24.192764 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.192744    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"]
Jul 28 13:08:24.196557 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.196540    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d01ec-predictor-8675b95b7b-r5bd2"]
Jul 28 13:08:24.344951 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:24.344922    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" path="/var/lib/kubelet/pods/fcf6dd94-3518-4850-8caf-1cfab68c8671/volumes"
Jul 28 13:08:28.984042 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:28.984001    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:29.176551 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:29.176515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:08:29.176991 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:29.176951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:08:31.461738 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.461705    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"]
Jul 28 13:08:31.462201 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.461961    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container" containerID="cri-o://4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6" gracePeriod=30
Jul 28 13:08:31.462201 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.462169    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kube-rbac-proxy" containerID="cri-o://a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac" gracePeriod=30
Jul 28 13:08:31.566777 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.566745    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"]
Jul 28 13:08:31.567147 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.567132    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kube-rbac-proxy"
Jul 28 13:08:31.567147 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.567149    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kube-rbac-proxy"
Jul 28 13:08:31.567262 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.567163    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container"
Jul 28 13:08:31.567262 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.567182    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container"
Jul 28 13:08:31.567262 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.567239    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kserve-container"
Jul 28 13:08:31.567262 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.567247    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf6dd94-3518-4850-8caf-1cfab68c8671" containerName="kube-rbac-proxy"
Jul 28 13:08:31.570533 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.570513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.573045 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.573024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-13bf1-kube-rbac-proxy-sar-config\""
Jul 28 13:08:31.573161 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.573056    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-13bf1-predictor-serving-cert\""
Jul 28 13:08:31.584552 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.584531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"]
Jul 28 13:08:31.668736 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.668695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32396bba-6723-4e61-a74c-16ee1b50628b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.668926 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.668803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32396bba-6723-4e61-a74c-16ee1b50628b-proxy-tls\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.668926 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.668899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsfs\" (UniqueName: \"kubernetes.io/projected/32396bba-6723-4e61-a74c-16ee1b50628b-kube-api-access-jrsfs\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.770209 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.770131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32396bba-6723-4e61-a74c-16ee1b50628b-proxy-tls\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.770209 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.770177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsfs\" (UniqueName: \"kubernetes.io/projected/32396bba-6723-4e61-a74c-16ee1b50628b-kube-api-access-jrsfs\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.770382 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.770214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32396bba-6723-4e61-a74c-16ee1b50628b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.770872 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.770844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32396bba-6723-4e61-a74c-16ee1b50628b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.772750 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.772721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32396bba-6723-4e61-a74c-16ee1b50628b-proxy-tls\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.779023 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.779002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsfs\" (UniqueName: \"kubernetes.io/projected/32396bba-6723-4e61-a74c-16ee1b50628b-kube-api-access-jrsfs\") pod \"error-404-isvc-13bf1-predictor-bbffd5856-2kmmm\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") " pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:31.881100 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:31.881073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:32.014467 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.014444    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"]
Jul 28 13:08:32.016941 ip-10-0-137-92 kubenswrapper[2563]: W0728 13:08:32.016913    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32396bba_6723_4e61_a74c_16ee1b50628b.slice/crio-939d0bf08ca79a5d7a3d0e998c1ed20dff9e8b78d95d282e70a503828396a403 WatchSource:0}: Error finding container 939d0bf08ca79a5d7a3d0e998c1ed20dff9e8b78d95d282e70a503828396a403: Status 404 returned error can't find the container with id 939d0bf08ca79a5d7a3d0e998c1ed20dff9e8b78d95d282e70a503828396a403
Jul 28 13:08:32.210870 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.210816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" event={"ID":"32396bba-6723-4e61-a74c-16ee1b50628b","Type":"ContainerStarted","Data":"eb4e652b79b510af25681447b4ba9f8004aba29b3925ccda0b59488a38308295"}
Jul 28 13:08:32.211069 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.210909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" event={"ID":"32396bba-6723-4e61-a74c-16ee1b50628b","Type":"ContainerStarted","Data":"addf2b496c3609c4a970768506d53f4be143c958e8168f0c8c9587f3c64a7353"}
Jul 28 13:08:32.211069 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.210943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:32.211069 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.210967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" event={"ID":"32396bba-6723-4e61-a74c-16ee1b50628b","Type":"ContainerStarted","Data":"939d0bf08ca79a5d7a3d0e998c1ed20dff9e8b78d95d282e70a503828396a403"}
Jul 28 13:08:32.211069 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.210996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:32.212515 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.212486    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:08:32.213013 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.212993    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerID="a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac" exitCode=2
Jul 28 13:08:32.213116 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.213059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" event={"ID":"9b966603-3307-401a-ae83-d8ae74c7c7b1","Type":"ContainerDied","Data":"a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac"}
Jul 28 13:08:32.228963 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.228924    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podStartSLOduration=1.228911789 podStartE2EDuration="1.228911789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:32.228059077 +0000 UTC m=+2020.446411788" watchObservedRunningTime="2026-07-28 13:08:32.228911789 +0000 UTC m=+2020.447264500"
Jul 28 13:08:32.877445 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:32.877404    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 28 13:08:33.217669 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:33.217631    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:08:33.984530 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:33.984496    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:33.984920 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:33.984592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:08:34.821453 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.821400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:08:34.896190 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.896158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cvn8\" (UniqueName: \"kubernetes.io/projected/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-api-access-5cvn8\") pod \"9b966603-3307-401a-ae83-d8ae74c7c7b1\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") "
Jul 28 13:08:34.896360 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.896210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-rbac-proxy-sar-config\") pod \"9b966603-3307-401a-ae83-d8ae74c7c7b1\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") "
Jul 28 13:08:34.896360 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.896265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b966603-3307-401a-ae83-d8ae74c7c7b1-proxy-tls\") pod \"9b966603-3307-401a-ae83-d8ae74c7c7b1\" (UID: \"9b966603-3307-401a-ae83-d8ae74c7c7b1\") "
Jul 28 13:08:34.896603 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.896578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b966603-3307-401a-ae83-d8ae74c7c7b1" (UID: "9b966603-3307-401a-ae83-d8ae74c7c7b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:34.898376 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.898348    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b966603-3307-401a-ae83-d8ae74c7c7b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b966603-3307-401a-ae83-d8ae74c7c7b1" (UID: "9b966603-3307-401a-ae83-d8ae74c7c7b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:34.898482 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.898411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-api-access-5cvn8" (OuterVolumeSpecName: "kube-api-access-5cvn8") pod "9b966603-3307-401a-ae83-d8ae74c7c7b1" (UID: "9b966603-3307-401a-ae83-d8ae74c7c7b1"). InnerVolumeSpecName "kube-api-access-5cvn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:34.997542 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.997508    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvn8\" (UniqueName: \"kubernetes.io/projected/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-api-access-5cvn8\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:34.997542 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.997534    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b966603-3307-401a-ae83-d8ae74c7c7b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:34.997542 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:34.997548    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b966603-3307-401a-ae83-d8ae74c7c7b1-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:35.226531 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.226499    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerID="4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6" exitCode=0
Jul 28 13:08:35.226684 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.226547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"
Jul 28 13:08:35.226684 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.226582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" event={"ID":"9b966603-3307-401a-ae83-d8ae74c7c7b1","Type":"ContainerDied","Data":"4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6"}
Jul 28 13:08:35.226684 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.226641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj" event={"ID":"9b966603-3307-401a-ae83-d8ae74c7c7b1","Type":"ContainerDied","Data":"3e8f032d1280079e26868b4eec423fa1303f9c246752f0f7c9faf081ace650d5"}
Jul 28 13:08:35.226684 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.226670    2563 scope.go:117] "RemoveContainer" containerID="a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac"
Jul 28 13:08:35.235553 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.235539    2563 scope.go:117] "RemoveContainer" containerID="4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6"
Jul 28 13:08:35.242843 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.242806    2563 scope.go:117] "RemoveContainer" containerID="a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac"
Jul 28 13:08:35.243098 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:08:35.243078    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac\": container with ID starting with a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac not found: ID does not exist" containerID="a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac"
Jul 28 13:08:35.243140 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.243107    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac"} err="failed to get container status \"a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac\": rpc error: code = NotFound desc = could not find container \"a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac\": container with ID starting with a4d988458fb087dd5764bff9dd3a8348010b5dac59b21d28a825f7e6c6cf15ac not found: ID does not exist"
Jul 28 13:08:35.243140 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.243125    2563 scope.go:117] "RemoveContainer" containerID="4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6"
Jul 28 13:08:35.243364 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:08:35.243347    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6\": container with ID starting with 4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6 not found: ID does not exist" containerID="4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6"
Jul 28 13:08:35.243405 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.243370    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6"} err="failed to get container status \"4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6\": rpc error: code = NotFound desc = could not find container \"4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6\": container with ID starting with 4e2530bcbf3dfdad76a07bde705668dacaa6b8605dfcba1dafa49c3e1bef3db6 not found: ID does not exist"
Jul 28 13:08:35.248778 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.248758    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"]
Jul 28 13:08:35.253380 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:35.253350    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4b3a0-predictor-5dc499d786-7lnlj"]
Jul 28 13:08:36.344538 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:36.344510    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" path="/var/lib/kubelet/pods/9b966603-3307-401a-ae83-d8ae74c7c7b1/volumes"
Jul 28 13:08:38.221795 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:38.221770    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:08:38.222282 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:38.222256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:08:38.984395 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:38.984354    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:39.177536 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:39.177503    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:08:43.983951 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:43.983904    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:48.222444 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:48.222405    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:08:48.983548 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:48.983506    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 13:08:49.177835 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:49.177788    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:08:50.869480 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:50.869452    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:08:50.939002 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:50.938973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls\") pod \"c1878b56-1edb-4f07-af9c-b7c807c95532\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") "
Jul 28 13:08:50.939144 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:50.939014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1878b56-1edb-4f07-af9c-b7c807c95532-openshift-service-ca-bundle\") pod \"c1878b56-1edb-4f07-af9c-b7c807c95532\" (UID: \"c1878b56-1edb-4f07-af9c-b7c807c95532\") "
Jul 28 13:08:50.939403 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:50.939379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1878b56-1edb-4f07-af9c-b7c807c95532-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c1878b56-1edb-4f07-af9c-b7c807c95532" (UID: "c1878b56-1edb-4f07-af9c-b7c807c95532"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:50.941193 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:50.941175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1878b56-1edb-4f07-af9c-b7c807c95532" (UID: "c1878b56-1edb-4f07-af9c-b7c807c95532"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:51.040056 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.039980    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1878b56-1edb-4f07-af9c-b7c807c95532-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:51.040056 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.040004    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1878b56-1edb-4f07-af9c-b7c807c95532-openshift-service-ca-bundle\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:08:51.298997 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.298915    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerID="6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19" exitCode=0
Jul 28 13:08:51.298997 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.298957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"
Jul 28 13:08:51.299205 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.298962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" event={"ID":"c1878b56-1edb-4f07-af9c-b7c807c95532","Type":"ContainerDied","Data":"6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19"}
Jul 28 13:08:51.299205 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.299064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b" event={"ID":"c1878b56-1edb-4f07-af9c-b7c807c95532","Type":"ContainerDied","Data":"f328141c5b93d57775788ed4e5c70e2dd8460df75902a7d17579b2243d1ccc45"}
Jul 28 13:08:51.299205 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.299087    2563 scope.go:117] "RemoveContainer" containerID="6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19"
Jul 28 13:08:51.307695 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.307678    2563 scope.go:117] "RemoveContainer" containerID="6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19"
Jul 28 13:08:51.307961 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:08:51.307943    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19\": container with ID starting with 6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19 not found: ID does not exist" containerID="6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19"
Jul 28 13:08:51.308022 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.307971    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19"} err="failed to get container status \"6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19\": rpc error: code = NotFound desc = could not find container \"6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19\": container with ID starting with 6da14e198a3c00f535daa9bedf403e517512870abfac4d613d6cb61711ee3d19 not found: ID does not exist"
Jul 28 13:08:51.320744 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.320716    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"]
Jul 28 13:08:51.325330 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:51.325309    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d01ec-59754655f9-2xm8b"]
Jul 28 13:08:52.345468 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:52.345428    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" path="/var/lib/kubelet/pods/c1878b56-1edb-4f07-af9c-b7c807c95532/volumes"
Jul 28 13:08:58.222683 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:58.222634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:08:59.177662 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:08:59.177622    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 13:09:08.222298 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:09:08.222256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:09:09.178281 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:09:09.178251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:09:18.223789 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:09:18.223709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:09:52.387675 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:09:52.387649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:09:52.396789 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:09:52.396767    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:14:52.413465 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:14:52.413438    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:14:52.424239 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:14:52.424217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:17:46.436171 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:46.436141    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"]
Jul 28 13:17:46.436742 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:46.436392    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" containerID="cri-o://addf2b496c3609c4a970768506d53f4be143c958e8168f0c8c9587f3c64a7353" gracePeriod=30
Jul 28 13:17:46.436742 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:46.436425    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kube-rbac-proxy" containerID="cri-o://eb4e652b79b510af25681447b4ba9f8004aba29b3925ccda0b59488a38308295" gracePeriod=30
Jul 28 13:17:46.642234 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:46.642201    2563 generic.go:356] "Generic (PLEG): container finished" podID="32396bba-6723-4e61-a74c-16ee1b50628b" containerID="eb4e652b79b510af25681447b4ba9f8004aba29b3925ccda0b59488a38308295" exitCode=2
Jul 28 13:17:46.642409 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:46.642279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" event={"ID":"32396bba-6723-4e61-a74c-16ee1b50628b","Type":"ContainerDied","Data":"eb4e652b79b510af25681447b4ba9f8004aba29b3925ccda0b59488a38308295"}
Jul 28 13:17:48.218695 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:48.218641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 28 13:17:48.223094 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:48.223066    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 13:17:49.656637 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.656605    2563 generic.go:356] "Generic (PLEG): container finished" podID="32396bba-6723-4e61-a74c-16ee1b50628b" containerID="addf2b496c3609c4a970768506d53f4be143c958e8168f0c8c9587f3c64a7353" exitCode=0
Jul 28 13:17:49.657054 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.656680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" event={"ID":"32396bba-6723-4e61-a74c-16ee1b50628b","Type":"ContainerDied","Data":"addf2b496c3609c4a970768506d53f4be143c958e8168f0c8c9587f3c64a7353"}
Jul 28 13:17:49.689619 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.689599    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:17:49.797751 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.797648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32396bba-6723-4e61-a74c-16ee1b50628b-kube-rbac-proxy-sar-config\") pod \"32396bba-6723-4e61-a74c-16ee1b50628b\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") "
Jul 28 13:17:49.797751 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.797696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32396bba-6723-4e61-a74c-16ee1b50628b-proxy-tls\") pod \"32396bba-6723-4e61-a74c-16ee1b50628b\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") "
Jul 28 13:17:49.797751 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.797719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrsfs\" (UniqueName: \"kubernetes.io/projected/32396bba-6723-4e61-a74c-16ee1b50628b-kube-api-access-jrsfs\") pod \"32396bba-6723-4e61-a74c-16ee1b50628b\" (UID: \"32396bba-6723-4e61-a74c-16ee1b50628b\") "
Jul 28 13:17:49.798154 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.798125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32396bba-6723-4e61-a74c-16ee1b50628b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32396bba-6723-4e61-a74c-16ee1b50628b" (UID: "32396bba-6723-4e61-a74c-16ee1b50628b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:17:49.799990 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.799967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32396bba-6723-4e61-a74c-16ee1b50628b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32396bba-6723-4e61-a74c-16ee1b50628b" (UID: "32396bba-6723-4e61-a74c-16ee1b50628b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:17:49.800073 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.800014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32396bba-6723-4e61-a74c-16ee1b50628b-kube-api-access-jrsfs" (OuterVolumeSpecName: "kube-api-access-jrsfs") pod "32396bba-6723-4e61-a74c-16ee1b50628b" (UID: "32396bba-6723-4e61-a74c-16ee1b50628b"). InnerVolumeSpecName "kube-api-access-jrsfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:17:49.899134 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.899093    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32396bba-6723-4e61-a74c-16ee1b50628b-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:17:49.899134 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.899129    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrsfs\" (UniqueName: \"kubernetes.io/projected/32396bba-6723-4e61-a74c-16ee1b50628b-kube-api-access-jrsfs\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:17:49.899134 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:49.899141    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32396bba-6723-4e61-a74c-16ee1b50628b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:17:50.662890 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:50.662785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm" event={"ID":"32396bba-6723-4e61-a74c-16ee1b50628b","Type":"ContainerDied","Data":"939d0bf08ca79a5d7a3d0e998c1ed20dff9e8b78d95d282e70a503828396a403"}
Jul 28 13:17:50.662890 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:50.662807    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"
Jul 28 13:17:50.663385 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:50.662909    2563 scope.go:117] "RemoveContainer" containerID="eb4e652b79b510af25681447b4ba9f8004aba29b3925ccda0b59488a38308295"
Jul 28 13:17:50.673496 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:50.673474    2563 scope.go:117] "RemoveContainer" containerID="addf2b496c3609c4a970768506d53f4be143c958e8168f0c8c9587f3c64a7353"
Jul 28 13:17:50.684315 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:50.684293    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"]
Jul 28 13:17:50.688878 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:50.688856    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-13bf1-predictor-bbffd5856-2kmmm"]
Jul 28 13:17:52.345969 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:17:52.345938    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" path="/var/lib/kubelet/pods/32396bba-6723-4e61-a74c-16ee1b50628b/volumes"
Jul 28 13:19:52.439332 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:19:52.439224    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:19:52.450241 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:19:52.450218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:24:52.471295 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:24:52.471267    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:24:52.486095 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:24:52.486072    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:25:40.428438 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:40.428398    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"]
Jul 28 13:25:40.428979 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:40.428627    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container" containerID="cri-o://9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4" gracePeriod=30
Jul 28 13:25:40.428979 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:40.428694    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kube-rbac-proxy" containerID="cri-o://4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7" gracePeriod=30
Jul 28 13:25:40.689526 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:40.689424    2563 generic.go:356] "Generic (PLEG): container finished" podID="507c16a1-485a-40ae-a166-4a0a8802217f" containerID="4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7" exitCode=2
Jul 28 13:25:40.689526 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:40.689487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" event={"ID":"507c16a1-485a-40ae-a166-4a0a8802217f","Type":"ContainerDied","Data":"4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7"}
Jul 28 13:25:43.578333 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.578309    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:25:43.689870 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.689765    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhb8g\" (UniqueName: \"kubernetes.io/projected/507c16a1-485a-40ae-a166-4a0a8802217f-kube-api-access-dhb8g\") pod \"507c16a1-485a-40ae-a166-4a0a8802217f\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") "
Jul 28 13:25:43.690037 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.689885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507c16a1-485a-40ae-a166-4a0a8802217f-kube-rbac-proxy-sar-config\") pod \"507c16a1-485a-40ae-a166-4a0a8802217f\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") "
Jul 28 13:25:43.690037 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.689923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507c16a1-485a-40ae-a166-4a0a8802217f-proxy-tls\") pod \"507c16a1-485a-40ae-a166-4a0a8802217f\" (UID: \"507c16a1-485a-40ae-a166-4a0a8802217f\") "
Jul 28 13:25:43.690263 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.690240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507c16a1-485a-40ae-a166-4a0a8802217f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "507c16a1-485a-40ae-a166-4a0a8802217f" (UID: "507c16a1-485a-40ae-a166-4a0a8802217f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:25:43.692184 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.692164    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507c16a1-485a-40ae-a166-4a0a8802217f-kube-api-access-dhb8g" (OuterVolumeSpecName: "kube-api-access-dhb8g") pod "507c16a1-485a-40ae-a166-4a0a8802217f" (UID: "507c16a1-485a-40ae-a166-4a0a8802217f"). InnerVolumeSpecName "kube-api-access-dhb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:25:43.692246 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.692208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507c16a1-485a-40ae-a166-4a0a8802217f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "507c16a1-485a-40ae-a166-4a0a8802217f" (UID: "507c16a1-485a-40ae-a166-4a0a8802217f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:25:43.703422 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.703395    2563 generic.go:356] "Generic (PLEG): container finished" podID="507c16a1-485a-40ae-a166-4a0a8802217f" containerID="9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4" exitCode=0
Jul 28 13:25:43.703572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.703454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"
Jul 28 13:25:43.703572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.703483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" event={"ID":"507c16a1-485a-40ae-a166-4a0a8802217f","Type":"ContainerDied","Data":"9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4"}
Jul 28 13:25:43.703572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.703544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85" event={"ID":"507c16a1-485a-40ae-a166-4a0a8802217f","Type":"ContainerDied","Data":"10817aa8f442558639674b8009f9df462386c7ecf160028f6e2971cd02f25d59"}
Jul 28 13:25:43.703572 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.703571    2563 scope.go:117] "RemoveContainer" containerID="4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7"
Jul 28 13:25:43.712266 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.712246    2563 scope.go:117] "RemoveContainer" containerID="9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4"
Jul 28 13:25:43.720030 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.720013    2563 scope.go:117] "RemoveContainer" containerID="4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7"
Jul 28 13:25:43.720277 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:25:43.720259    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7\": container with ID starting with 4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7 not found: ID does not exist" containerID="4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7"
Jul 28 13:25:43.720322 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.720285    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7"} err="failed to get container status \"4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7\": rpc error: code = NotFound desc = could not find container \"4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7\": container with ID starting with 4efb71d6a87e9f6178ee22a396fab9ff17c0d676b181d1f55880282f869e32c7 not found: ID does not exist"
Jul 28 13:25:43.720322 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.720301    2563 scope.go:117] "RemoveContainer" containerID="9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4"
Jul 28 13:25:43.720526 ip-10-0-137-92 kubenswrapper[2563]: E0728 13:25:43.720511    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4\": container with ID starting with 9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4 not found: ID does not exist" containerID="9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4"
Jul 28 13:25:43.720563 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.720532    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4"} err="failed to get container status \"9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4\": rpc error: code = NotFound desc = could not find container \"9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4\": container with ID starting with 9aefcfb91e73b600bca8eaf08fd64f301a0df44fbeff8d84feba5fcb8b7e11e4 not found: ID does not exist"
Jul 28 13:25:43.725697 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.725675    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"]
Jul 28 13:25:43.731774 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.731749    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dde97-predictor-db9fd4f9b-7kx85"]
Jul 28 13:25:43.791534 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.791502    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507c16a1-485a-40ae-a166-4a0a8802217f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:25:43.791534 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.791530    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507c16a1-485a-40ae-a166-4a0a8802217f-proxy-tls\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:25:43.791711 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:43.791543    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhb8g\" (UniqueName: \"kubernetes.io/projected/507c16a1-485a-40ae-a166-4a0a8802217f-kube-api-access-dhb8g\") on node \"ip-10-0-137-92.ec2.internal\" DevicePath \"\""
Jul 28 13:25:44.344209 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:25:44.344175    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" path="/var/lib/kubelet/pods/507c16a1-485a-40ae-a166-4a0a8802217f/volumes"
Jul 28 13:26:14.986966 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:14.986928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dh4h9_30fd4493-5528-42f8-b77e-a73c763a2116/global-pull-secret-syncer/0.log"
Jul 28 13:26:15.137444 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:15.137412    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cptn6_d62d21db-38dd-451a-967d-91a67fa38c39/konnectivity-agent/0.log"
Jul 28 13:26:15.254705 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:15.254602    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-92.ec2.internal_e0ede8a7a9a486a43a28960227e9b07d/haproxy/0.log"
Jul 28 13:26:19.117246 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.117203    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/alertmanager/0.log"
Jul 28 13:26:19.147591 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.147559    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/config-reloader/0.log"
Jul 28 13:26:19.172080 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.172054    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/kube-rbac-proxy-web/0.log"
Jul 28 13:26:19.200473 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.200444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/kube-rbac-proxy/0.log"
Jul 28 13:26:19.222579 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.222550    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/kube-rbac-proxy-metric/0.log"
Jul 28 13:26:19.246385 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.246366    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/prom-label-proxy/0.log"
Jul 28 13:26:19.272750 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.272723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b9cdd796-2a87-4572-9b01-c876f0bd161f/init-config-reloader/0.log"
Jul 28 13:26:19.311760 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.311730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-q9znf_d14de7a2-2981-41a0-b3c2-980fd5ba681b/cluster-monitoring-operator/0.log"
Jul 28 13:26:19.432462 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.432389    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5694649-m945p_7cc71eb8-0f26-446b-9581-6f801391c2c1/metrics-server/0.log"
Jul 28 13:26:19.663785 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.663748    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtxlb_42b85603-415a-425d-a57f-85fdcbb354de/node-exporter/0.log"
Jul 28 13:26:19.689965 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.689878    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtxlb_42b85603-415a-425d-a57f-85fdcbb354de/kube-rbac-proxy/0.log"
Jul 28 13:26:19.710389 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:19.710363    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtxlb_42b85603-415a-425d-a57f-85fdcbb354de/init-textfile/0.log"
Jul 28 13:26:20.006408 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:20.006378    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-rt5m6_a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4/prometheus-operator/0.log"
Jul 28 13:26:20.028515 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:20.028490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-rt5m6_a9ac5e55-bd18-4a6f-aaa1-2fd7811034b4/kube-rbac-proxy/0.log"
Jul 28 13:26:22.607203 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:22.607124    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-jqxjb_1383c340-e6e9-4605-82eb-959f8b778bd7/download-server/0.log"
Jul 28 13:26:23.540185 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540153    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"]
Jul 28 13:26:23.540473 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540460    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540475    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540487    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540492    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540502    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540508    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540514    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container"
Jul 28 13:26:23.540519 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540519    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540531    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540544    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540549    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540556    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540611    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kserve-container"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540618    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kserve-container"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b966603-3307-401a-ae83-d8ae74c7c7b1" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1878b56-1edb-4f07-af9c-b7c807c95532" containerName="sequence-graph-d01ec"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540637    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540642    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32396bba-6723-4e61-a74c-16ee1b50628b" containerName="kserve-container"
Jul 28 13:26:23.540735 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.540648    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="507c16a1-485a-40ae-a166-4a0a8802217f" containerName="kube-rbac-proxy"
Jul 28 13:26:23.543869 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.543850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.546236 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.546212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6jls2\"/\"kube-root-ca.crt\""
Jul 28 13:26:23.546356 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.546293    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6jls2\"/\"openshift-service-ca.crt\""
Jul 28 13:26:23.547111 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.547093    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6jls2\"/\"default-dockercfg-gtdpd\""
Jul 28 13:26:23.551483 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.551461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"]
Jul 28 13:26:23.633772 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.633732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-proc\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.633772 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.633772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrtk\" (UniqueName: \"kubernetes.io/projected/7489456f-e2fa-4e3c-9272-fb9c8c160f21-kube-api-access-xjrtk\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.634268 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.633909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-podres\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.634268 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.633938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-lib-modules\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.634268 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.633995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-sys\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.734781 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-podres\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.734781 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-lib-modules\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-sys\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-lib-modules\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-podres\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-sys\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.734988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-proc\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.735028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrtk\" (UniqueName: \"kubernetes.io/projected/7489456f-e2fa-4e3c-9272-fb9c8c160f21-kube-api-access-xjrtk\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.735117 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.735046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7489456f-e2fa-4e3c-9272-fb9c8c160f21-proc\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.743574 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.743545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrtk\" (UniqueName: \"kubernetes.io/projected/7489456f-e2fa-4e3c-9272-fb9c8c160f21-kube-api-access-xjrtk\") pod \"perf-node-gather-daemonset-j576j\" (UID: \"7489456f-e2fa-4e3c-9272-fb9c8c160f21\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.855067 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.854974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:23.989443 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:23.989405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"]
Jul 28 13:26:23.992906 ip-10-0-137-92 kubenswrapper[2563]: W0728 13:26:23.992877    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7489456f_e2fa_4e3c_9272_fb9c8c160f21.slice/crio-d5d607acc4efb1f012729d873f63a79afb91901f9ad500e3f9f48610b4484d41 WatchSource:0}: Error finding container d5d607acc4efb1f012729d873f63a79afb91901f9ad500e3f9f48610b4484d41: Status 404 returned error can't find the container with id d5d607acc4efb1f012729d873f63a79afb91901f9ad500e3f9f48610b4484d41
Jul 28 13:26:24.092489 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.092448    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clr4g_9f34d2d2-3bbb-4f64-95e4-e2087909f7cd/dns/0.log"
Jul 28 13:26:24.113513 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.113414    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clr4g_9f34d2d2-3bbb-4f64-95e4-e2087909f7cd/kube-rbac-proxy/0.log"
Jul 28 13:26:24.225336 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.225310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vz9w_71572878-eed8-4109-913e-eee642504dba/dns-node-resolver/0.log"
Jul 28 13:26:24.756953 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.756923    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wrq8_5cd0dcaa-46fa-4da7-a97e-c8737a1bf049/node-ca/0.log"
Jul 28 13:26:24.888723 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.888688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j" event={"ID":"7489456f-e2fa-4e3c-9272-fb9c8c160f21","Type":"ContainerStarted","Data":"6d323c8f08ac19b07b7a41cf24d8baf20704578313b33f225adfdfe1c75b108b"}
Jul 28 13:26:24.888926 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.888742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j" event={"ID":"7489456f-e2fa-4e3c-9272-fb9c8c160f21","Type":"ContainerStarted","Data":"d5d607acc4efb1f012729d873f63a79afb91901f9ad500e3f9f48610b4484d41"}
Jul 28 13:26:24.888926 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.888838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:24.906217 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:24.906161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j" podStartSLOduration=1.90614625 podStartE2EDuration="1.90614625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:24.90419844 +0000 UTC m=+3093.122551152" watchObservedRunningTime="2026-07-28 13:26:24.90614625 +0000 UTC m=+3093.124498996"
Jul 28 13:26:25.920285 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:25.920230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jnrmn_18018de2-4df9-4e7b-bbe9-6aab7abf990a/serve-healthcheck-canary/0.log"
Jul 28 13:26:26.413598 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:26.413568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bb42n_cb20ff7e-9864-4d09-b996-6a4f5e5c76ad/kube-rbac-proxy/0.log"
Jul 28 13:26:26.433741 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:26.433711    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bb42n_cb20ff7e-9864-4d09-b996-6a4f5e5c76ad/exporter/0.log"
Jul 28 13:26:26.454401 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:26.454373    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bb42n_cb20ff7e-9864-4d09-b996-6a4f5e5c76ad/extractor/0.log"
Jul 28 13:26:28.780696 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:28.780664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-qhbmp_3f980bfb-c1f5-46e3-99cc-970589383c71/manager/0.log"
Jul 28 13:26:28.798766 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:28.798741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2t7m7_52594f10-5bd2-4687-b357-b0038becdca8/server/0.log"
Jul 28 13:26:30.903720 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:30.903695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-j576j"
Jul 28 13:26:32.765591 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:32.765544    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-h2drb_cc47e803-bc15-49c7-9d65-2ad5ca5cd890/migrator/0.log"
Jul 28 13:26:32.792716 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:32.792689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-h2drb_cc47e803-bc15-49c7-9d65-2ad5ca5cd890/graceful-termination/0.log"
Jul 28 13:26:34.231626 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.231598    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/kube-multus-additional-cni-plugins/0.log"
Jul 28 13:26:34.253285 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.253256    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/egress-router-binary-copy/0.log"
Jul 28 13:26:34.273556 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.273532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/cni-plugins/0.log"
Jul 28 13:26:34.294424 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.294388    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/bond-cni-plugin/0.log"
Jul 28 13:26:34.317597 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.317571    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/routeoverride-cni/0.log"
Jul 28 13:26:34.337994 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.337960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/whereabouts-cni-bincopy/0.log"
Jul 28 13:26:34.361485 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.361457    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9gc9r_e4eebd04-e2b5-449b-843f-dd9f5dd1f9c6/whereabouts-cni/0.log"
Jul 28 13:26:34.767779 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.767741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z5bdt_f8db7ce2-f506-453a-8d17-36c233734369/kube-multus/0.log"
Jul 28 13:26:34.829840 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.829722    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dqk86_7ae2f41f-73c2-451c-b9fe-c6a94c59a238/network-metrics-daemon/0.log"
Jul 28 13:26:34.848945 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:34.848917    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dqk86_7ae2f41f-73c2-451c-b9fe-c6a94c59a238/kube-rbac-proxy/0.log"
Jul 28 13:26:35.902378 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.902351    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-controller/0.log"
Jul 28 13:26:35.919319 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.919290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/1.log"
Jul 28 13:26:35.919477 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.919290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovn-acl-logging/0.log"
Jul 28 13:26:35.937656 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.937629    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/kube-rbac-proxy-node/0.log"
Jul 28 13:26:35.959993 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.959964    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 13:26:35.978665 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.978600    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/northd/0.log"
Jul 28 13:26:35.999084 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:35.999058    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/nbdb/0.log"
Jul 28 13:26:36.024866 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:36.024847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/sbdb/0.log"
Jul 28 13:26:36.072926 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:36.072898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7tvz_b7b35311-e67c-403b-afd8-7755a95f64a5/ovnkube-controller/0.log"
Jul 28 13:26:37.316206 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:37.316173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5tsz9_76dd0f1f-9e03-4e9c-8e32-456992cda116/network-check-target-container/0.log"
Jul 28 13:26:38.265280 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:38.265253    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-57pb8_343e89fd-3c76-4dcd-8154-d28f1ba2daec/iptables-alerter/0.log"
Jul 28 13:26:39.002145 ip-10-0-137-92 kubenswrapper[2563]: I0728 13:26:39.002115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mw6vl_bb855a11-5572-4be0-a3ba-a0830feedecf/tuned/0.log"