Jul 23 00:21:23.792032 ip-10-0-139-138 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 23 00:21:23.792041 ip-10-0-139-138 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 00:21:23.792048 ip-10-0-139-138 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 23 00:21:23.792290 ip-10-0-139-138 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 23 00:21:33.887804 ip-10-0-139-138 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 23 00:21:33.887845 ip-10-0-139-138 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1109d769e752402ababa10f1086a92a7 --
Jul 23 00:23:26.090381 ip-10-0-139-138 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.434447    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439687    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439700    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439704    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439708    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439710    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439715    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439719    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:23:26.529793 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439722    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:23:26.499886 ip-10-0-139-138 systemd[1]: Started Kubernetes Kubelet.
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439725    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439729    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439732    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439735    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439739    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439741    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439744    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439748    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439751    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439754    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439757    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439760    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439762    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439765    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439768    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439771    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439774    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439777    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439780    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:23:26.532813 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439782    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439785    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439788    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439790    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439793    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439795    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439798    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439801    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439804    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439806    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439809    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439812    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439814    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439817    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439820    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439823    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439825    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439828    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439831    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:23:26.535146 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439833    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439835    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439838    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439841    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439843    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439846    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439848    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439850    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439853    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439856    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439858    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439860    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439863    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439865    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439868    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439870    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439872    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439876    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439879    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439882    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:23:26.537868 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439884    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439888    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439891    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439893    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439896    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439899    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439902    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439904    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439907    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439910    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439913    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439916    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439918    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439921    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439923    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439926    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439928    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439930    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439933    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439936    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:23:26.544311 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439938    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439941    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439944    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439948    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439952    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439954    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439957    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439959    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439962    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439965    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439968    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439971    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.439973    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440682    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440689    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440692    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440694    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440698    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440701    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:23:26.546655 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440703    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440706    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440709    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440712    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440714    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440717    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440719    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440722    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440725    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440728    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440730    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440733    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440736    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440738    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440741    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440744    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440747    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440749    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440752    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440754    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:23:26.551775 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440757    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440759    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440762    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440764    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440767    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440770    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440772    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440774    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440777    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440780    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440782    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440785    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440788    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440791    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440794    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440796    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440800    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440803    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440805    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440809    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:23:26.552922 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440811    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440814    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440816    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440819    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440822    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440824    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440827    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440829    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440832    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440835    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440837    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440840    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440842    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440844    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440847    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440849    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440868    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440872    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440875    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440878    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:23:26.554819 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440881    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440884    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440887    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440891    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440894    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440897    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440900    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440903    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440906    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440909    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440912    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440914    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440917    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440921    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440925    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440939    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440943    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440945    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440948    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:23:26.556064 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440951    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440956    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440960    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440963    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440965    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440968    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440973    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440977    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440981    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440984    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440987    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440990    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440993    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.440996    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441417    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441423    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441426    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441429    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441433    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:23:26.561058 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441437    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441440    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441443    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441445    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441448    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441450    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441453    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441455    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441458    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441460    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441463    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441465    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441468    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441472    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441475    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441477    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441480    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441483    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441485    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441489    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:23:26.561695 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441492    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441494    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441497    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441499    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441502    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441504    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441506    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441509    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441512    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441514    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441517    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441519    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441522    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441525    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441527    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441530    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441532    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441534    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441537    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441540    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:23:26.562938 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441542    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441545    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441548    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441550    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441552    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441556    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441560    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441562    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441565    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441567    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441570    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441574    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441577    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441579    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441582    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441584    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441587    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441589    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441591    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441594    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:23:26.563571 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441596    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441600    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441604    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441607    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441609    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441613    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441615    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441617    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441620    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441622    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441625    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441627    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441630    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441633    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441635    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441637    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441641    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441645    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441647    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:23:26.564415 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441651    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441654    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441657    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441660    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441664    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441667    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441669    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441672    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441675    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441678    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441682    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441685    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441688    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441691    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.441693    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441759    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441765    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441771    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441776    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441781    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441784    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 00:23:26.640255 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441788    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441793    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441796    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441799    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441802    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441805    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441808    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441811    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441814    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441819    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441822    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441825    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441829    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441832    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441835    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441840    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441843    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441847    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441850    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441853    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441857    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441860    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441863    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441866    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 00:23:26.640998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441869    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441872    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441881    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441884    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441887    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441890    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441893    2569 flags.go:64] FLAG: --enable-server="true"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441896    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441901    2569 flags.go:64] FLAG: --event-burst="100"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441904    2569 flags.go:64] FLAG: --event-qps="50"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441907    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441910    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441913    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441916    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441919    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441922    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441925    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441929    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441932    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441935    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441938    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441941    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441944    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441948    2569 flags.go:64] FLAG: --feature-gates=""
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441951    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 00:23:26.641830 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441954    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441957    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441960    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441964    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441967    2569 flags.go:64] FLAG: --help="false"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441969    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-139-138.ec2.internal"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441972    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441975    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441978    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441981    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441985    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441987    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441990    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441993    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.441996    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442000    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442003    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442005    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442008    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442011    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442014    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442017    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442020    2569 flags.go:64] FLAG: --lock-file=""
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442023    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 00:23:26.643278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442027    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442030    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442035    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442038    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442041    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442044    2569 flags.go:64] FLAG: --logging-format="text"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442048    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442051    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442054    2569 flags.go:64] FLAG: --manifest-url=""
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442057    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442061    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442064    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442081    2569 flags.go:64] FLAG: --max-pods="110"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442084    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442087    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442090    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442093    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442096    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442099    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442103    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442109    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442112    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442115    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 00:23:26.644327 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442118    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442128    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442134    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442137    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442140    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442143    2569 flags.go:64] FLAG: --port="10250"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442146    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442149    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08814130aba0a8af7"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442152    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442155    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442159    2569 flags.go:64] FLAG: --register-node="true"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442162    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442170    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442173    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442176    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442180    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442184    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442187    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442190    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442193    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442196    2569 flags.go:64] FLAG: --runonce="false"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442198    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442201    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442204    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442207    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 00:23:26.644984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442210    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442213    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442216    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442219    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442221    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442224    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442227    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442230    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442233    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442237    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442240    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442245    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442248    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442251    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442255    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442258    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442261    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442265    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442268    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442271    2569 flags.go:64] FLAG: --v="2"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442275    2569 flags.go:64] FLAG: --version="false"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442279    2569 flags.go:64] FLAG: --vmodule=""
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442283    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442287    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442412    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:23:26.645822 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442416    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442421    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442425    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442428    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442431    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442433    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442436    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442439    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442442    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442445    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442448    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442450    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442453    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442456    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442458    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442461    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442463    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442466    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442469    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442471    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:23:26.646512 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442474    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442477    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442479    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442482    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442486    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442488    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442491    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442493    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442496    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442500    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442503    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442505    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442508    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442510    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442513    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442516    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442518    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442521    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442523    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442525    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:23:26.647144 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442528    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442531    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442533    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442536    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442538    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442541    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442543    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442546    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442548    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442551    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442554    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442556    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442559    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442561    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442564    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442566    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442570    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442573    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442575    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:23:26.647711 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442578    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442580    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442584    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442587    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442590    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442592    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442595    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442597    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442600    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442602    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442605    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442609    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442612    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442615    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442618    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442621    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442624    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442627    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442630    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442632    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:23:26.648324 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442635    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442638    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442641    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442643    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442646    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442650    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442652    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442655    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442658    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442663    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442665    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442668    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442671    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442674    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442680    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442683    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442686    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442689    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:23:26.648871 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.442692    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.442698    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.449427    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.449444    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.450120    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.450130    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450188    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450194    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450197    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450200    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450204    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450207    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450209    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450212    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450215    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:23:26.649382 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450218    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450221    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450223    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450226    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450229    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450232    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450238    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450240    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450243    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450246    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450248    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450251    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450253    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450256    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450259    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450261    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450266    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450269    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450272    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:23:26.649829 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450276    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450279    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450282    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450284    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450287    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450290    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450292    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450295    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450297    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450300    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450302    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450305    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450307    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450310    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450312    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450316    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450318    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450321    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450324    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450328    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:23:26.650417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450332    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450336    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450339    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450341    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450344    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450346    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450349    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450352    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450355    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450357    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450360    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450364    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450367    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450369    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450374    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450377    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450380    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450383    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450386    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:23:26.650963 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450389    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450392    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450394    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450397    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450400    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450402    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450405    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450407    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450410    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450412    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450415    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450417    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450421    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450424    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450426    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450428    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450431    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450434    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450437    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450439    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:23:26.651483 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450442    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450444    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450447    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450449    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450454    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450456    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450459    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450461    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450464    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450467    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450470    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450472    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.450477    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450588    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450593    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:23:26.652095 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450596    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450599    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450602    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450605    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450608    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450611    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450614    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450616    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450619    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450622    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450627    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450630    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450633    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450635    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450638    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450640    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450644    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450648    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450651    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:23:26.652510 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450654    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450657    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450661    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450664    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450667    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450669    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450672    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450675    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450678    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450680    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450683    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450685    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450689    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450692    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450695    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450698    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450701    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450704    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450707    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450710    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:23:26.653402 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450712    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450715    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450718    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450722    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450725    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450727    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450730    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450733    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450735    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450738    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450740    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450743    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450745    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450748    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450751    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450754    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450757    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450759    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450762    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450764    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:23:26.654030 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450767    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450770    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450772    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450775    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450778    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450780    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450782    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450785    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450788    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450790    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450792    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450795    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450798    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450800    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450803    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450807    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450810    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450812    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450815    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450818    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:23:26.654659 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450820    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450823    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450825    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450828    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450831    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450834    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450837    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450841    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450843    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450846    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450848    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450852    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450855    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450858    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450860    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450863    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450869    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:23:26.655925 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:26.450872    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.450877    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.450980    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.453123    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.454629    2569 server.go:1019] "Starting client certificate rotation"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.454742    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.454790    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.458610    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.460061    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.469763    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.471989    2569 log.go:25] "Validated CRI v1 image API"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.472743    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 00:23:26.656552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.475226    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a0cad9a-d678-4a5f-b50f-25c8927b11d3:/dev/nvme0n1p4 9f2a8369-7b16-4611-922b-15aecc9fcb0e:/dev/nvme0n1p3]
Jul 23 00:23:26.658026 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.475241    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.479964    2569 manager.go:216] Machine: {Timestamp:2026-07-23 00:23:26.478872103 +0000 UTC m=+0.274751346 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099899 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cfb83b8bd01f6641a4abdd228b428 SystemUUID:ec2cfb83-b8bd-01f6-641a-4abdd228b428 BootID:1109d769-e752-402a-baba-10f1086a92a7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9d:54:f8:e2:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9d:54:f8:e2:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:8b:f4:4f:ce:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.480083    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.480167    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.480596    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.481629    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.481659    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 00:23:26.742181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.482280    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.482290    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.482299    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.482830    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.484088    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.487964    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488094    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488121    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488130    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488147    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488156    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488165    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488522    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.488539    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.490682    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.493467    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.493496    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494565    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494580    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494587    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494592    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494601    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494609    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494618    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494624    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494630    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494636    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494644    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.494650    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.495224    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.495235    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.498604    2569 server.go:1268] "Started kubelet"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.498682    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.498795    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.498699    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.499882    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.499901    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.499981    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.500450    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 00:23:26.769403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.501094    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.503662    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.504910    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-138.ec2.internal.18c4c368a1781ace  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-138.ec2.internal,UID:ip-10-0-139-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-138.ec2.internal,},FirstTimestamp:2026-07-23 00:23:26.498577102 +0000 UTC m=+0.294456343,LastTimestamp:2026-07-23 00:23:26.498577102 +0000 UTC m=+0.294456343,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-138.ec2.internal,}"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.505821    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-flcr5"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.507602    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.507618    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.507718    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508146    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508174    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508188    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508264    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508272    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508581    2569 factory.go:55] Registering systemd factory
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508594    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508860    2569 factory.go:153] Registering CRI-O factory
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508874    2569 factory.go:223] Registration of the crio container factory successfully
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508923    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508959    2569 factory.go:103] Registering Raw factory
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.508972    2569 manager.go:1196] Started watching for new ooms in manager
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.509469    2569 manager.go:349] Starting recovery of all containers
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.511006    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.511133    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.514407    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-flcr5"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.520103    2569 manager.go:354] Recovery completed
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.523439    2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.529658    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.531565    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.531585    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.531595    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.532244    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.532252    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.532265    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 00:23:26.770847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.532581    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.533328    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.533357    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.533374    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.533768    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-138.ec2.internal.18c4c368a36fa1e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-138.ec2.internal,UID:ip-10-0-139-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-138.ec2.internal,},FirstTimestamp:2026-07-23 00:23:26.531576293 +0000 UTC m=+0.327455535,LastTimestamp:2026-07-23 00:23:26.531576293 +0000 UTC m=+0.327455535,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-138.ec2.internal,}"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.534397    2569 policy_none.go:49] "None policy: Start"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.534414    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.534424    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.536417    2569 policy_none.go:47] "Start"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.608239    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.629819    2569 manager.go:337] "Starting Device Plugin manager"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.630008    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.630019    2569 server.go:87] "Starting device plugin registration server"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.630340    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.630354    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.630451    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.630544    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.630555    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.631181    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.631242    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.717542    2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-138.ec2.internal\" not found" node="ip-10-0-139-138.ec2.internal"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.730626    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.737834    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.739136    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.739174    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.739199    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.739238    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.741621    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.746091    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.746127    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.746142    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.746168    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-138.ec2.internal"
Jul 23 00:23:26.772117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.753762    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-138.ec2.internal"
Jul 23 00:23:26.773184 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.753779    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-138.ec2.internal\": node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:26.773184 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.771926    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.839609    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"]
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.839707    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.840746    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.840773    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.840783    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.843238    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.843371    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.843401    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.844009    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.844030    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.844040    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.844061    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.844043    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.844089    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.846399    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.846424    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.847122    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.847142    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:23:26.852861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:26.847154    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:23:26.935123 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.872710    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.033701 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.968351    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-138.ec2.internal\" not found" node="ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.033701 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.973126    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.033701 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:26.973281    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-138.ec2.internal\" not found" node="ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.033701 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.019069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/428a120582b292f1d0cc24c60d6d6dd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal\" (UID: \"428a120582b292f1d0cc24c60d6d6dd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.033701 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.019131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428a120582b292f1d0cc24c60d6d6dd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal\" (UID: \"428a120582b292f1d0cc24c60d6d6dd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.033701 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.019158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc72966d35a3fd5b5406ca4a255a917-config\") pod \"kube-apiserver-proxy-ip-10-0-139-138.ec2.internal\" (UID: \"6cc72966d35a3fd5b5406ca4a255a917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.073634    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.119417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/428a120582b292f1d0cc24c60d6d6dd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal\" (UID: \"428a120582b292f1d0cc24c60d6d6dd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.119450    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428a120582b292f1d0cc24c60d6d6dd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal\" (UID: \"428a120582b292f1d0cc24c60d6d6dd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.119469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc72966d35a3fd5b5406ca4a255a917-config\") pod \"kube-apiserver-proxy-ip-10-0-139-138.ec2.internal\" (UID: \"6cc72966d35a3fd5b5406ca4a255a917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.119488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/428a120582b292f1d0cc24c60d6d6dd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal\" (UID: \"428a120582b292f1d0cc24c60d6d6dd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.119510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/428a120582b292f1d0cc24c60d6d6dd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal\" (UID: \"428a120582b292f1d0cc24c60d6d6dd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.138750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.119495    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cc72966d35a3fd5b5406ca4a255a917-config\") pod \"kube-apiserver-proxy-ip-10-0-139-138.ec2.internal\" (UID: \"6cc72966d35a3fd5b5406ca4a255a917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.255481 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.173828    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.355661 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.274878    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.355661 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.274900    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.355661 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.277223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.355661 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.332346    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:23:27.462977 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.377962    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.462977 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.455221    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 00:23:27.462977 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.455404    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 00:23:27.462977 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.456500    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 00:23:27.478948 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.478922    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.523016 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.522974    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 00:18:27 +0000 UTC" deadline="2028-02-06 17:51:46.577738484 +0000 UTC"
Jul 23 00:23:27.523016 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.523006    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13529h28m19.054735309s"
Jul 23 00:23:27.579281 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.579228    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.672440 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.672412    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:23:27.679708 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:27.679686    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-138.ec2.internal\" not found"
Jul 23 00:23:27.741256 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.741173    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:23:27.809426 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.809368    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.821446 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.821418    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 00:23:27.822292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.822273    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal"
Jul 23 00:23:27.832046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.832021    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 00:23:27.989331 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:27.989299    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428a120582b292f1d0cc24c60d6d6dd9.slice/crio-26ab4182ba0e04ddd497080b64b798ee5002482c2d930399e8107a0fb7d79410 WatchSource:0}: Error finding container 26ab4182ba0e04ddd497080b64b798ee5002482c2d930399e8107a0fb7d79410: Status 404 returned error can't find the container with id 26ab4182ba0e04ddd497080b64b798ee5002482c2d930399e8107a0fb7d79410
Jul 23 00:23:27.989669 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:27.989647    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc72966d35a3fd5b5406ca4a255a917.slice/crio-89985935d29f737ef6ad6302ccdaf3efbb0f12c9e25687572844b0a76cf22848 WatchSource:0}: Error finding container 89985935d29f737ef6ad6302ccdaf3efbb0f12c9e25687572844b0a76cf22848: Status 404 returned error can't find the container with id 89985935d29f737ef6ad6302ccdaf3efbb0f12c9e25687572844b0a76cf22848
Jul 23 00:23:27.994761 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:27.994746    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:23:28.489305 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.489218    2569 apiserver.go:52] "Watching apiserver"
Jul 23 00:23:28.499750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.499710    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 00:23:28.500147 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.500099    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bvppm","kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ct5pv","openshift-image-registry/node-ca-6tccw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal","openshift-multus/multus-29gkg","openshift-multus/network-metrics-daemon-p9dgl","openshift-network-operator/iptables-alerter-vb98q","kube-system/konnectivity-agent-jbdtg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf","openshift-dns/node-resolver-dl9s5","openshift-multus/multus-additional-cni-plugins-5gnx8","openshift-network-diagnostics/network-check-target-h74t5","openshift-ovn-kubernetes/ovnkube-node-nmfpw"]
Jul 23 00:23:28.502854 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.502824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:28.502969 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.502946    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:28.505192 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.505167    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.507739 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.507706    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.508249 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.508177    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fwwbr\""
Jul 23 00:23:28.508249 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.508233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.508383 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.508186    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.508533 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.508508    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 00:23:28.510643 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.510621    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6zsrf\""
Jul 23 00:23:28.510776 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.510671    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.510948 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.510927    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.511046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.511008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.513560 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.513351    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.513560 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.513359    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.513560 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.513535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.513966 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.513944    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 00:23:28.514046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.513978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2kxk6\""
Jul 23 00:23:28.515718 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.515529    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 00:23:28.515718 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.515705    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.515838 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.515821    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 00:23:28.516278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.516257    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29frj\""
Jul 23 00:23:28.516386 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.516281    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.516386 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.516372    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:23:28.516498 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.516463    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 00:23:28.516924 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.516607    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 00:23:28.516924 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.516830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.519354 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.519193    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 00:23:28.519454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.519399    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.519705 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.519567    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cghjk\""
Jul 23 00:23:28.519705 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.519606    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 00:23:28.519969 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.519874    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.520067 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.520009    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.521004 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.520987    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 00:23:28.521857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.521831    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 00:23:28.521956 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.521940    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 00:23:28.522218 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.522200    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:28.522301 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.522237    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fxg4c\""
Jul 23 00:23:28.522419 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.522386    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:28.524808 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.524786    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.524906 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.524884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:28.524961 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.524917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysconfig\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.524961 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.524941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa1702e8-c63c-4224-b143-454913151172-host\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.525045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.524967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z82bj\" (UniqueName: \"kubernetes.io/projected/aa1702e8-c63c-4224-b143-454913151172-kube-api-access-z82bj\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.525045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gz5q\" (UniqueName: \"kubernetes.io/projected/d0d5d50e-1891-4bba-bf96-30fff80caffd-kube-api-access-6gz5q\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:28.525136 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f49d87a-dd38-40f7-9d59-526873d06917-tmp-dir\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.525136 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59fb8\" (UniqueName: \"kubernetes.io/projected/2f49d87a-dd38-40f7-9d59-526873d06917-kube-api-access-59fb8\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.525136 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-modprobe-d\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-kubernetes\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysctl-conf\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f49d87a-dd38-40f7-9d59-526873d06917-hosts-file\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.525292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-systemd\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-sys\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-tuned\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525327    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg99n\" (UniqueName: \"kubernetes.io/projected/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-kube-api-access-tg99n\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525351    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa1702e8-c63c-4224-b143-454913151172-serviceca\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysctl-d\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-run\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-lib-modules\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-var-lib-kubelet\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525506    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-host\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.525539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.525527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-tmp\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.527210 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.527185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.527726 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.527686    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-54zb7\""
Jul 23 00:23:28.527812 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.527708    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.527812 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.527793    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.528131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.528115    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 00:23:28.530502 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.530382    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 00:23:28.530502 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.530458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.531499 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.531471    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dcx9m\""
Jul 23 00:23:28.532534 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.532486    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 00:23:28.533361 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.533326    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:28.535661 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.535606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 00:23:28.536476 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.536418    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.539018 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.538823    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 00:23:28.539018 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.538837    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cgck4\""
Jul 23 00:23:28.539018 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.539003    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 00:23:28.541401 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.541378    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmlft"
Jul 23 00:23:28.547281 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.547131    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmlft"
Jul 23 00:23:28.610003 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.609970    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 00:23:28.626461 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626425    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-device-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.626610 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-cni-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.626610 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-netns\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.626610 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626567    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-slash\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.626610 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-systemd\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-ovnkube-config\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9691368-fa76-4ebe-8ad7-5b79e74bf346-agent-certs\") pod \"konnectivity-agent-jbdtg\" (UID: \"a9691368-fa76-4ebe-8ad7-5b79e74bf346\") " pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa1702e8-c63c-4224-b143-454913151172-host\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/336c34d5-f2af-4441-8fe9-51d93d0f1202-host-slash\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-system-cni-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.626840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626817    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa1702e8-c63c-4224-b143-454913151172-host\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46njf\" (UniqueName: \"kubernetes.io/projected/7ef7466d-00eb-4a44-972c-a48ac18eecf5-kube-api-access-46njf\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-cnibin\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59fb8\" (UniqueName: \"kubernetes.io/projected/2f49d87a-dd38-40f7-9d59-526873d06917-kube-api-access-59fb8\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-kubelet\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-env-overrides\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.626981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-sys\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627017    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-socket-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-etc-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-sys\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627115    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9691368-fa76-4ebe-8ad7-5b79e74bf346-konnectivity-ca\") pod \"konnectivity-agent-jbdtg\" (UID: \"a9691368-fa76-4ebe-8ad7-5b79e74bf346\") " pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-lib-modules\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw88h\" (UniqueName: \"kubernetes.io/projected/336c34d5-f2af-4441-8fe9-51d93d0f1202-kube-api-access-cw88h\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.627243 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-cni-bin\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/280511d0-25a2-4136-8252-1b3ece74cc94-ovn-node-metrics-cert\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-lib-modules\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-k8s-cni-cncf-io\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627435    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.627476    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f49d87a-dd38-40f7-9d59-526873d06917-tmp-dir\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627509    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-systemd\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.627560    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:23:29.127537084 +0000 UTC m=+2.923416337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627579    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-sys-fs\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627597    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-systemd\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-cnibin\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.627955 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627636    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-log-socket\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b914057-5d8f-4393-93d5-389ab25f66ad-kubelet-config\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627698    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-var-lib-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysctl-d\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-host\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-tmp\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2m2x\" (UniqueName: \"kubernetes.io/projected/280511d0-25a2-4136-8252-1b3ece74cc94-kube-api-access-p2m2x\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ef7466d-00eb-4a44-972c-a48ac18eecf5-cni-binary-copy\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-cni-bin\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.627998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gz5q\" (UniqueName: \"kubernetes.io/projected/d0d5d50e-1891-4bba-bf96-30fff80caffd-kube-api-access-6gz5q\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysctl-conf\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-host\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.628717 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628087    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f49d87a-dd38-40f7-9d59-526873d06917-tmp-dir\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysctl-d\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa1702e8-c63c-4224-b143-454913151172-serviceca\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbx7\" (UniqueName: \"kubernetes.io/projected/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-kube-api-access-rmbx7\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-os-release\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f49d87a-dd38-40f7-9d59-526873d06917-hosts-file\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-hostroot\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628423    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-ovn\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-node-log\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-cni-netd\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.629454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-run\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-var-lib-kubelet\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628583    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/336c34d5-f2af-4441-8fe9-51d93d0f1202-iptables-alerter-script\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-run-netns\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f49d87a-dd38-40f7-9d59-526873d06917-hosts-file\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-multus-certs\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysconfig\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82bj\" (UniqueName: \"kubernetes.io/projected/aa1702e8-c63c-4224-b143-454913151172-kube-api-access-z82bj\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b914057-5d8f-4393-93d5-389ab25f66ad-dbus\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-conf-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628751    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-run\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa1702e8-c63c-4224-b143-454913151172-serviceca\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628770    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-daemon-config\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-var-lib-kubelet\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630214 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-system-cni-dir\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysconfig\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-modprobe-d\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-kubernetes\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.628983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-modprobe-d\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg99n\" (UniqueName: \"kubernetes.io/projected/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-kube-api-access-tg99n\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-kubernetes\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-cni-multus\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-etc-kubernetes\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629239    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25wq\" (UniqueName: \"kubernetes.io/projected/aee7a54d-abce-4798-9b79-bd89f42aae36-kube-api-access-s25wq\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-kubelet\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-sysctl-conf\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.630922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-tuned\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.631770 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-registration-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.631770 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-socket-dir-parent\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.631770 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-os-release\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.631770 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-systemd-units\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.631770 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.629444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-ovnkube-script-lib\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.632921 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.632894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-tmp\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.633071 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.633052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-etc-tuned\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.640618 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.640594    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59fb8\" (UniqueName: \"kubernetes.io/projected/2f49d87a-dd38-40f7-9d59-526873d06917-kube-api-access-59fb8\") pod \"node-resolver-dl9s5\" (UID: \"2f49d87a-dd38-40f7-9d59-526873d06917\") " pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.643266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.643244    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gz5q\" (UniqueName: \"kubernetes.io/projected/d0d5d50e-1891-4bba-bf96-30fff80caffd-kube-api-access-6gz5q\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:28.643397 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.643292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82bj\" (UniqueName: \"kubernetes.io/projected/aa1702e8-c63c-4224-b143-454913151172-kube-api-access-z82bj\") pod \"node-ca-6tccw\" (UID: \"aa1702e8-c63c-4224-b143-454913151172\") " pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.643397 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.643381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg99n\" (UniqueName: \"kubernetes.io/projected/d5f5c0ff-9cb9-483a-ad1e-591e07fea234-kube-api-access-tg99n\") pod \"tuned-ct5pv\" (UID: \"d5f5c0ff-9cb9-483a-ad1e-591e07fea234\") " pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.730640 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.730604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-conf-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.730827 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.730665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-daemon-config\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.730827 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.730695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-system-cni-dir\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.730827 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.730694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-conf-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.731001 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.730812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-system-cni-dir\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.731227 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.731199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.731305 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.731263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.731305 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.731300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-cni-multus\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.731426 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.731408    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-cni-multus\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.731848 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.731822    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-daemon-config\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.732000 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.731982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-etc-selinux\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.732069 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.732237 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-etc-kubernetes\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.732298 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25wq\" (UniqueName: \"kubernetes.io/projected/aee7a54d-abce-4798-9b79-bd89f42aae36-kube-api-access-s25wq\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.732363 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-kubelet\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732363 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-registration-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.732479 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-socket-dir-parent\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.732479 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-os-release\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.732479 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-systemd-units\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-ovnkube-script-lib\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-device-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.732631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-cni-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.732631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-netns\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.732631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-slash\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-systemd\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-ovnkube-config\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732768    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9691368-fa76-4ebe-8ad7-5b79e74bf346-agent-certs\") pod \"konnectivity-agent-jbdtg\" (UID: \"a9691368-fa76-4ebe-8ad7-5b79e74bf346\") " pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/336c34d5-f2af-4441-8fe9-51d93d0f1202-host-slash\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-system-cni-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.732888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46njf\" (UniqueName: \"kubernetes.io/projected/7ef7466d-00eb-4a44-972c-a48ac18eecf5-kube-api-access-46njf\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-cnibin\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-kubelet\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.732990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-env-overrides\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-socket-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-etc-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9691368-fa76-4ebe-8ad7-5b79e74bf346-konnectivity-ca\") pod \"konnectivity-agent-jbdtg\" (UID: \"a9691368-fa76-4ebe-8ad7-5b79e74bf346\") " pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw88h\" (UniqueName: \"kubernetes.io/projected/336c34d5-f2af-4441-8fe9-51d93d0f1202-kube-api-access-cw88h\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733224    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.733266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733257    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-cni-bin\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/280511d0-25a2-4136-8252-1b3ece74cc94-ovn-node-metrics-cert\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-k8s-cni-cncf-io\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-sys-fs\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-cnibin\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-log-socket\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733533    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b914057-5d8f-4393-93d5-389ab25f66ad-kubelet-config\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-var-lib-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2m2x\" (UniqueName: \"kubernetes.io/projected/280511d0-25a2-4136-8252-1b3ece74cc94-kube-api-access-p2m2x\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ef7466d-00eb-4a44-972c-a48ac18eecf5-cni-binary-copy\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.733759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-cni-bin\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733817    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbx7\" (UniqueName: \"kubernetes.io/projected/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-kube-api-access-rmbx7\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-os-release\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-hostroot\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.733999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-ovn\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-node-log\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-cni-netd\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/336c34d5-f2af-4441-8fe9-51d93d0f1202-iptables-alerter-script\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-run-netns\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-multus-certs\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.734328 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b914057-5d8f-4393-93d5-389ab25f66ad-dbus\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.734922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734459    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b914057-5d8f-4393-93d5-389ab25f66ad-dbus\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.734922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734520    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-etc-kubernetes\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.734922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-kubelet\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.735074 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.734952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-registration-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.735074 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-socket-dir-parent\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.735185 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.735185 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-os-release\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.735291 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735186    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-k8s-cni-cncf-io\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.735291 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.735248    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:28.735291 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735254    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-sys-fs\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.735427 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-cnibin\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.735427 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.735320    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:23:29.23530029 +0000 UTC m=+3.031179536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:28.735427 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-log-socket\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.735593 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735431    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b914057-5d8f-4393-93d5-389ab25f66ad-kubelet-config\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:28.735593 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-hostroot\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.735747 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.735718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-var-lib-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.736109 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.736080    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.736216 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.736192    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-run-netns\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739048 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.736243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aee7a54d-abce-4798-9b79-bd89f42aae36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.736255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-multus-certs\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.736360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-cni-netd\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.736891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/336c34d5-f2af-4441-8fe9-51d93d0f1202-iptables-alerter-script\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ef7466d-00eb-4a44-972c-a48ac18eecf5-cni-binary-copy\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-os-release\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737244    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-cni-bin\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-systemd-units\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739187 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737451    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-cnibin\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-device-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-run-netns\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.737823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-kubelet\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738127    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-multus-cni-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aee7a54d-abce-4798-9b79-bd89f42aae36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-slash\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738229    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-etc-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738389    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9691368-fa76-4ebe-8ad7-5b79e74bf346-konnectivity-ca\") pod \"konnectivity-agent-jbdtg\" (UID: \"a9691368-fa76-4ebe-8ad7-5b79e74bf346\") " pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-ovnkube-script-lib\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-env-overrides\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-systemd\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/336c34d5-f2af-4441-8fe9-51d93d0f1202-host-slash\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.739658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/280511d0-25a2-4136-8252-1b3ece74cc94-ovnkube-config\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-system-cni-dir\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738901    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-openvswitch\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738908    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ef7466d-00eb-4a44-972c-a48ac18eecf5-host-var-lib-cni-bin\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-node-log\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/280511d0-25a2-4136-8252-1b3ece74cc94-run-ovn\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.738994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-socket-dir\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.740439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.739406    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/280511d0-25a2-4136-8252-1b3ece74cc94-ovn-node-metrics-cert\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.741848 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.741821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9691368-fa76-4ebe-8ad7-5b79e74bf346-agent-certs\") pod \"konnectivity-agent-jbdtg\" (UID: \"a9691368-fa76-4ebe-8ad7-5b79e74bf346\") " pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.750804 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.750747    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal" event={"ID":"428a120582b292f1d0cc24c60d6d6dd9","Type":"ContainerStarted","Data":"26ab4182ba0e04ddd497080b64b798ee5002482c2d930399e8107a0fb7d79410"}
Jul 23 00:23:28.752144 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.752118    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:23:28.752294 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.752148    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:23:28.752294 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.752164    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:28.752294 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:28.752280    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:23:29.252227632 +0000 UTC m=+3.048106880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:28.753775 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.753705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal" event={"ID":"6cc72966d35a3fd5b5406ca4a255a917","Type":"ContainerStarted","Data":"89985935d29f737ef6ad6302ccdaf3efbb0f12c9e25687572844b0a76cf22848"}
Jul 23 00:23:28.756168 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.756102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2m2x\" (UniqueName: \"kubernetes.io/projected/280511d0-25a2-4136-8252-1b3ece74cc94-kube-api-access-p2m2x\") pod \"ovnkube-node-nmfpw\" (UID: \"280511d0-25a2-4136-8252-1b3ece74cc94\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.756299 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.756273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw88h\" (UniqueName: \"kubernetes.io/projected/336c34d5-f2af-4441-8fe9-51d93d0f1202-kube-api-access-cw88h\") pod \"iptables-alerter-vb98q\" (UID: \"336c34d5-f2af-4441-8fe9-51d93d0f1202\") " pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.756363 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.756320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbx7\" (UniqueName: \"kubernetes.io/projected/ec4d00d5-bd22-42dd-a4fd-888fe1f59826-kube-api-access-rmbx7\") pod \"aws-ebs-csi-driver-node-f2plf\" (UID: \"ec4d00d5-bd22-42dd-a4fd-888fe1f59826\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.756791 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.756767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46njf\" (UniqueName: \"kubernetes.io/projected/7ef7466d-00eb-4a44-972c-a48ac18eecf5-kube-api-access-46njf\") pod \"multus-29gkg\" (UID: \"7ef7466d-00eb-4a44-972c-a48ac18eecf5\") " pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.757486 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.757460    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25wq\" (UniqueName: \"kubernetes.io/projected/aee7a54d-abce-4798-9b79-bd89f42aae36-kube-api-access-s25wq\") pod \"multus-additional-cni-plugins-5gnx8\" (UID: \"aee7a54d-abce-4798-9b79-bd89f42aae36\") " pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.818988 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.818948    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dl9s5"
Jul 23 00:23:28.828812 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.828784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ct5pv"
Jul 23 00:23:28.831288 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.831249    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f49d87a_dd38_40f7_9d59_526873d06917.slice/crio-d87e76260617d71ee5a8e5a8368ae1f76ef1714082e16e207e3326ce69e63054 WatchSource:0}: Error finding container d87e76260617d71ee5a8e5a8368ae1f76ef1714082e16e207e3326ce69e63054: Status 404 returned error can't find the container with id d87e76260617d71ee5a8e5a8368ae1f76ef1714082e16e207e3326ce69e63054
Jul 23 00:23:28.839031 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.839002    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f5c0ff_9cb9_483a_ad1e_591e07fea234.slice/crio-84312f6789ec844e9524e2e4b4f8fdb583ee25357e37d07d6900b081eb8dc51c WatchSource:0}: Error finding container 84312f6789ec844e9524e2e4b4f8fdb583ee25357e37d07d6900b081eb8dc51c: Status 404 returned error can't find the container with id 84312f6789ec844e9524e2e4b4f8fdb583ee25357e37d07d6900b081eb8dc51c
Jul 23 00:23:28.841813 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.841656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6tccw"
Jul 23 00:23:28.848018 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.847995    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:28.852274 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.852242    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1702e8_c63c_4224_b143_454913151172.slice/crio-7ad58f7afe63126a3f9665c5df0ef352d4c0993ab78301baff2fdbbf5ddd68e7 WatchSource:0}: Error finding container 7ad58f7afe63126a3f9665c5df0ef352d4c0993ab78301baff2fdbbf5ddd68e7: Status 404 returned error can't find the container with id 7ad58f7afe63126a3f9665c5df0ef352d4c0993ab78301baff2fdbbf5ddd68e7
Jul 23 00:23:28.855723 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.855522    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29gkg"
Jul 23 00:23:28.864918 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.864894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:28.866145 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.866111    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef7466d_00eb_4a44_972c_a48ac18eecf5.slice/crio-2f312ccd4bdb36f955110b28f9b6e889e584947a5bc3b317a6466689a9971fca WatchSource:0}: Error finding container 2f312ccd4bdb36f955110b28f9b6e889e584947a5bc3b317a6466689a9971fca: Status 404 returned error can't find the container with id 2f312ccd4bdb36f955110b28f9b6e889e584947a5bc3b317a6466689a9971fca
Jul 23 00:23:28.874480 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.874457    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vb98q"
Jul 23 00:23:28.876417 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.876391    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9691368_fa76_4ebe_8ad7_5b79e74bf346.slice/crio-a9daaed3ebab16cffd1f1a3c61c078dd14259b403005216720d17836ca62d8c4 WatchSource:0}: Error finding container a9daaed3ebab16cffd1f1a3c61c078dd14259b403005216720d17836ca62d8c4: Status 404 returned error can't find the container with id a9daaed3ebab16cffd1f1a3c61c078dd14259b403005216720d17836ca62d8c4
Jul 23 00:23:28.883600 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.883551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf"
Jul 23 00:23:28.884753 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.884729    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336c34d5_f2af_4441_8fe9_51d93d0f1202.slice/crio-405f6df121310a39f1b159727cd9df4e2b0da419e57cf79a03d3a75aa4027c04 WatchSource:0}: Error finding container 405f6df121310a39f1b159727cd9df4e2b0da419e57cf79a03d3a75aa4027c04: Status 404 returned error can't find the container with id 405f6df121310a39f1b159727cd9df4e2b0da419e57cf79a03d3a75aa4027c04
Jul 23 00:23:28.890146 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:28.889933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5gnx8"
Jul 23 00:23:28.896362 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.896335    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4d00d5_bd22_42dd_a4fd_888fe1f59826.slice/crio-e17640ac0aae60bae0a689a2800a4065984c10afdb91d623e26706403a85d193 WatchSource:0}: Error finding container e17640ac0aae60bae0a689a2800a4065984c10afdb91d623e26706403a85d193: Status 404 returned error can't find the container with id e17640ac0aae60bae0a689a2800a4065984c10afdb91d623e26706403a85d193
Jul 23 00:23:28.904308 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:23:28.904283    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee7a54d_abce_4798_9b79_bd89f42aae36.slice/crio-c855f25db2fb25bb7c5d3f4f104255e1971da838ec3ed4560ab1bbb622ccedcc WatchSource:0}: Error finding container c855f25db2fb25bb7c5d3f4f104255e1971da838ec3ed4560ab1bbb622ccedcc: Status 404 returned error can't find the container with id c855f25db2fb25bb7c5d3f4f104255e1971da838ec3ed4560ab1bbb622ccedcc
Jul 23 00:23:29.138285 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.138187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:29.138461 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.138387    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:29.138517 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.138493    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:23:30.138453446 +0000 UTC m=+3.934332680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:29.238781 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.238745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:29.238952 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.238873    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:29.238952 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.238928    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:23:30.238914374 +0000 UTC m=+4.034793604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:29.339549 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.339511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:29.339764 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.339747    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:23:29.339842 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.339774    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:23:29.339842 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.339789    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:29.339942 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:29.339862    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:23:30.339841273 +0000 UTC m=+4.135720527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:29.461849 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.461762    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:23:29.550828 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.550771    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:18:28 +0000 UTC" deadline="2028-04-07 06:27:22.02534197 +0000 UTC"
Jul 23 00:23:29.550828 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.550813    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14982h3m52.47453485s"
Jul 23 00:23:29.757415 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.757321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vb98q" event={"ID":"336c34d5-f2af-4441-8fe9-51d93d0f1202","Type":"ContainerStarted","Data":"405f6df121310a39f1b159727cd9df4e2b0da419e57cf79a03d3a75aa4027c04"}
Jul 23 00:23:29.758615 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.758588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jbdtg" event={"ID":"a9691368-fa76-4ebe-8ad7-5b79e74bf346","Type":"ContainerStarted","Data":"a9daaed3ebab16cffd1f1a3c61c078dd14259b403005216720d17836ca62d8c4"}
Jul 23 00:23:29.759761 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.759736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29gkg" event={"ID":"7ef7466d-00eb-4a44-972c-a48ac18eecf5","Type":"ContainerStarted","Data":"2f312ccd4bdb36f955110b28f9b6e889e584947a5bc3b317a6466689a9971fca"}
Jul 23 00:23:29.760956 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.760927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ct5pv" event={"ID":"d5f5c0ff-9cb9-483a-ad1e-591e07fea234","Type":"ContainerStarted","Data":"84312f6789ec844e9524e2e4b4f8fdb583ee25357e37d07d6900b081eb8dc51c"}
Jul 23 00:23:29.762489 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.762441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerStarted","Data":"c855f25db2fb25bb7c5d3f4f104255e1971da838ec3ed4560ab1bbb622ccedcc"}
Jul 23 00:23:29.764219 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.764195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf" event={"ID":"ec4d00d5-bd22-42dd-a4fd-888fe1f59826","Type":"ContainerStarted","Data":"e17640ac0aae60bae0a689a2800a4065984c10afdb91d623e26706403a85d193"}
Jul 23 00:23:29.766099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.766075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"a64ab4a8c70730b0fe63e55fdadbef2663b8d286cba51dab7d02672671404601"}
Jul 23 00:23:29.770687 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.770659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6tccw" event={"ID":"aa1702e8-c63c-4224-b143-454913151172","Type":"ContainerStarted","Data":"7ad58f7afe63126a3f9665c5df0ef352d4c0993ab78301baff2fdbbf5ddd68e7"}
Jul 23 00:23:29.773451 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:29.773424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dl9s5" event={"ID":"2f49d87a-dd38-40f7-9d59-526873d06917","Type":"ContainerStarted","Data":"d87e76260617d71ee5a8e5a8368ae1f76ef1714082e16e207e3326ce69e63054"}
Jul 23 00:23:30.146659 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.146573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:30.146828 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.146755    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:30.146828 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.146812    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:23:32.146794495 +0000 UTC m=+5.942673726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:30.247215 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.247059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:30.247386 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.247294    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:30.247386 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.247351    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:23:32.247333558 +0000 UTC m=+6.043212803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:30.348092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.348055    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:30.348341 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.348306    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:23:30.348341 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.348334    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:23:30.348476 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.348348    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:30.348476 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.348411    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:23:32.348392826 +0000 UTC m=+6.144272061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:30.551822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.551776    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:18:28 +0000 UTC" deadline="2028-02-11 19:43:26.125475916 +0000 UTC"
Jul 23 00:23:30.551822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.551821    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13651h19m55.573660132s"
Jul 23 00:23:30.741302 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.741218    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:30.741460 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.741355    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:30.741739 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.741545    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:30.741739 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:30.741222    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:30.741739 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.741683    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:30.741959 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:30.741734    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:31.794249 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:31.793755    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal" event={"ID":"6cc72966d35a3fd5b5406ca4a255a917","Type":"ContainerStarted","Data":"dbb5bf62a9d31c0944568ddceae7394f51ae095a4e1407e269c7446249c7237c"}
Jul 23 00:23:31.801546 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:31.800883    2569 generic.go:356] "Generic (PLEG): container finished" podID="428a120582b292f1d0cc24c60d6d6dd9" containerID="7783393cf9dd0e3507a87ce2f7516dc9ba294e4950bdeb1454fe3e3ef2b807b0" exitCode=0
Jul 23 00:23:31.801546 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:31.800929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal" event={"ID":"428a120582b292f1d0cc24c60d6d6dd9","Type":"ContainerDied","Data":"7783393cf9dd0e3507a87ce2f7516dc9ba294e4950bdeb1454fe3e3ef2b807b0"}
Jul 23 00:23:31.825595 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:31.825533    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-138.ec2.internal" podStartSLOduration=3.825511074 podStartE2EDuration="3.825511074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:31.810438981 +0000 UTC m=+5.606318236" watchObservedRunningTime="2026-07-23 00:23:31.825511074 +0000 UTC m=+5.621390330"
Jul 23 00:23:32.173101 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:32.172992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:32.173281 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.173163    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:32.173281 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.173230    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:23:36.173210601 +0000 UTC m=+9.969089847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:32.274453 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:32.274365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:32.274645 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.274555    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:32.274645 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.274641    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:23:36.274617635 +0000 UTC m=+10.070496869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:32.375741 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:32.375692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:32.375973 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.375954    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:23:32.376054 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.375982    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:23:32.376054 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.375998    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:32.376218 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.376063    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:23:36.376044719 +0000 UTC m=+10.171923955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:32.741115 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:32.740241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:32.741115 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.740380    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:32.741115 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:32.740824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:32.741115 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.740936    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:32.741115 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:32.740999    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:32.741115 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:32.741072    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:34.741250 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:34.740478    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:34.741250 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:34.740626    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:34.741250 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:34.741006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:34.741250 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:34.741133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:34.741250 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:34.741182    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:34.741250 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:34.741252    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:36.212238 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:36.212194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:36.212699 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.212357    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:36.212699 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.212443    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:23:44.212420577 +0000 UTC m=+18.008299823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:36.313080 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:36.313014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:36.313242 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.313167    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:36.313242 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.313230    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:23:44.313212485 +0000 UTC m=+18.109091718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:36.413773 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:36.413739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:36.413931 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.413878    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:23:36.413931 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.413897    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:23:36.413931 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.413908    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:36.414118 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.413970    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:23:44.41395382 +0000 UTC m=+18.209833049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:36.741109 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:36.740985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:36.741304 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.741121    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:36.741736 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:36.741460    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:36.741736 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.741597    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:36.741736 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:36.741633    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:36.741736 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:36.741701    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:38.739798 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:38.739759    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:38.740277 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:38.739767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:38.740277 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:38.739902    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:38.740277 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:38.739767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:38.740277 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:38.739965    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:38.740277 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:38.740044    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:40.740212 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:40.740178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:40.740663 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:40.740178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:40.740663 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:40.740310    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:40.740663 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:40.740183    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:40.740663 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:40.740375    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:40.740663 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:40.740473    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:42.740153 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:42.740116    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:42.740153 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:42.740150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:42.740714 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:42.740116    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:42.740714 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:42.740251    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:42.740714 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:42.740383    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:42.740714 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:42.740545    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:44.271645 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:44.271603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:44.272031 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.271746    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:44.272031 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.271815    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:24:00.271797949 +0000 UTC m=+34.067677178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:23:44.372315 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:44.372271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:44.372479 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.372439    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:44.372533 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.372519    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:24:00.372500574 +0000 UTC m=+34.168379826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:23:44.472876 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:44.472837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:44.473093 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.472944    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:23:44.473093 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.472963    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:23:44.473093 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.472972    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:44.473093 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.473028    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:00.473014109 +0000 UTC m=+34.268893349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:23:44.740044 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:44.740007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:44.740226 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:44.740051    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:44.740226 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:44.740007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:44.740226 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.740131    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:44.740226 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.740212    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:44.740456 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:44.740289    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:46.740647 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:46.740611    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:46.741167 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:46.740724    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:46.741167 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:46.740745    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:46.741167 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:46.740787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:46.741167 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:46.740863    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:46.741167 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:46.741015    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:48.739639 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:48.739606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:48.740099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:48.739606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:48.740099 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:48.739735    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:48.740099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:48.739609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:48.740099 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:48.739843    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:48.740099 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:48.739870    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:49.842014 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.841785    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842373    2569 generic.go:356] "Generic (PLEG): container finished" podID="280511d0-25a2-4136-8252-1b3ece74cc94" containerID="114e5eb442589c8505ec0e1094a51ef0f3521f9a09445541fdcc2a0c3c9c58a9" exitCode=1
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"5cb56842fb058f2cb5e70c2411acc6e6993ac23b776f7e2e7e042d32fe28041a"}
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"663593488b1959c0ea2bcc4525689da504a6cd31a6d3b7a5df33dec1fafd0c4b"}
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842628    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"1aee076be7af2f4c47684184f5c0d7e567dc752890d943b713dff2dc270a5f25"}
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"c4a6d6977c17bd60bcce46c25583e56b57d3015cf54a039a77549fbf3375bcea"}
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerDied","Data":"114e5eb442589c8505ec0e1094a51ef0f3521f9a09445541fdcc2a0c3c9c58a9"}
Jul 23 00:23:49.842855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.842775    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"86086672770a8401602f3b20939d9c239ab5b4a5fc47aa357a2e2d1249f8f3e5"}
Jul 23 00:23:49.845349 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.844984    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6tccw" event={"ID":"aa1702e8-c63c-4224-b143-454913151172","Type":"ContainerStarted","Data":"7f8863dfb79cade99107b1729635372f5900f87ae1e857a1974221264619dd84"}
Jul 23 00:23:49.847611 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.847583    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dl9s5" event={"ID":"2f49d87a-dd38-40f7-9d59-526873d06917","Type":"ContainerStarted","Data":"4273f287dbf24e75db34ffaac3f518335ca2b236778f7b27bc2f7d7c4569348e"}
Jul 23 00:23:49.849849 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.849814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal" event={"ID":"428a120582b292f1d0cc24c60d6d6dd9","Type":"ContainerStarted","Data":"a4c18ff1986962f405d5792fd32114466cf7dd223744dd0b2f70177cb33eaf51"}
Jul 23 00:23:49.851248 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.851223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jbdtg" event={"ID":"a9691368-fa76-4ebe-8ad7-5b79e74bf346","Type":"ContainerStarted","Data":"6f375fface9137c93f40ddc90419cb95a9bad8a98d6722d0c942583b092ed7a4"}
Jul 23 00:23:49.853036 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.853008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29gkg" event={"ID":"7ef7466d-00eb-4a44-972c-a48ac18eecf5","Type":"ContainerStarted","Data":"66b9c54389559dfc08c7bd0b2ebe04ee4b238f79eba889652afe4e61cc3e3817"}
Jul 23 00:23:49.855309 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.855285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ct5pv" event={"ID":"d5f5c0ff-9cb9-483a-ad1e-591e07fea234","Type":"ContainerStarted","Data":"997269511ba0ba5a2b1cca9f98e2ad27813ac444252cfd33247ecfd04d8b3329"}
Jul 23 00:23:49.857123 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.857100    2569 generic.go:356] "Generic (PLEG): container finished" podID="aee7a54d-abce-4798-9b79-bd89f42aae36" containerID="b9ec9845ef3f74c02961b1f90d9fb12b2776cc4985eaac5ab3399ba2eeed4bc8" exitCode=0
Jul 23 00:23:49.857231 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.857181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerDied","Data":"b9ec9845ef3f74c02961b1f90d9fb12b2776cc4985eaac5ab3399ba2eeed4bc8"}
Jul 23 00:23:49.858994 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.858976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf" event={"ID":"ec4d00d5-bd22-42dd-a4fd-888fe1f59826","Type":"ContainerStarted","Data":"3e6d43ce15d481c929704a0c4844feff4a1e37f9cf07441d51978c3895d571ae"}
Jul 23 00:23:49.862870 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.862817    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6tccw" podStartSLOduration=2.9538348450000003 podStartE2EDuration="22.862768743s" totalImagesPullingTime="19.908933898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:49.862697928 +0000 UTC m=+23.658577194" watchObservedRunningTime="2026-07-23 00:23:49.862768743 +0000 UTC m=+23.658648020"
Jul 23 00:23:49.878099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.878004    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jbdtg" podStartSLOduration=7.442779257 podStartE2EDuration="22.877986047s" totalImagesPullingTime="15.43520679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:49.877510827 +0000 UTC m=+23.673390078" watchObservedRunningTime="2026-07-23 00:23:49.877986047 +0000 UTC m=+23.673865299"
Jul 23 00:23:49.891347 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.891304    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dl9s5" podStartSLOduration=2.965670625 podStartE2EDuration="22.891293246s" totalImagesPullingTime="19.925622621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:49.89104584 +0000 UTC m=+23.686925091" watchObservedRunningTime="2026-07-23 00:23:49.891293246 +0000 UTC m=+23.687172496"
Jul 23 00:23:49.907226 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.907187    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-29gkg" podStartSLOduration=2.9896803419999998 podStartE2EDuration="22.907175112s" totalImagesPullingTime="19.91749477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:49.906891158 +0000 UTC m=+23.702770409" watchObservedRunningTime="2026-07-23 00:23:49.907175112 +0000 UTC m=+23.703054363"
Jul 23 00:23:49.942571 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:49.942519    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-138.ec2.internal" podStartSLOduration=21.942500312 podStartE2EDuration="21.942500312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:49.942339831 +0000 UTC m=+23.738219083" watchObservedRunningTime="2026-07-23 00:23:49.942500312 +0000 UTC m=+23.738379566"
Jul 23 00:23:50.681168 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.681142    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 00:23:50.740329 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.739919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:50.740329 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.739919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:50.740329 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:50.740090    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:50.740329 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.740120    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:50.740329 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:50.740254    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:50.740329 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:50.740334    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:50.863986 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.863948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vb98q" event={"ID":"336c34d5-f2af-4441-8fe9-51d93d0f1202","Type":"ContainerStarted","Data":"721c242bb10f6acfcb8c6b4f65a55f5ca52b1b91074eae90e35fabcaf06cce34"}
Jul 23 00:23:50.866135 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.866111    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf" event={"ID":"ec4d00d5-bd22-42dd-a4fd-888fe1f59826","Type":"ContainerStarted","Data":"c78662c53f1381b6bdc812d055a2fc6671a0ec63014c92550379022db65e32e4"}
Jul 23 00:23:50.873412 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.873391    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:50.873936 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.873918    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:50.877534 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.877495    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vb98q" podStartSLOduration=4.003739442 podStartE2EDuration="23.877481525s" totalImagesPullingTime="19.873742083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:50.877297301 +0000 UTC m=+24.673176551" watchObservedRunningTime="2026-07-23 00:23:50.877481525 +0000 UTC m=+24.673360777"
Jul 23 00:23:50.878091 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:50.878045    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ct5pv" podStartSLOduration=3.955737703 podStartE2EDuration="23.878036623s" totalImagesPullingTime="19.92229892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:49.961274343 +0000 UTC m=+23.757153590" watchObservedRunningTime="2026-07-23 00:23:50.878036623 +0000 UTC m=+24.673915873"
Jul 23 00:23:51.642980 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:51.642860    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T00:23:50.681161235Z","UUID":"2e97192f-36e6-4b94-9712-95615e6aaeeb","Handler":null,"Name":"","Endpoint":""}
Jul 23 00:23:51.644600 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:51.644559    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 00:23:51.644749 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:51.644609    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 00:23:51.868451 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:51.868420    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:51.868950 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:51.868931    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jbdtg"
Jul 23 00:23:52.740205 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.739947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:52.740416 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.739953    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:52.740416 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:52.740240    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:52.740416 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:52.740322    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:52.740416 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.740001    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:52.740584 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:52.740418    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:52.875143 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.875107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf" event={"ID":"ec4d00d5-bd22-42dd-a4fd-888fe1f59826","Type":"ContainerStarted","Data":"696dfe7b4b30a214146f20379312e65fcd2a86f07fad110d672542dab6a88cb5"}
Jul 23 00:23:52.881404 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.881381    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:23:52.881868 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.881837    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"be0934db4204a2cbd61d3edf4dda7c8088a684f05f081ff694e67bf64d13aba0"}
Jul 23 00:23:52.896526 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:52.896481    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f2plf" podStartSLOduration=2.768879638 podStartE2EDuration="25.896467539s" totalImagesPullingTime="23.127587901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:52.895959966 +0000 UTC m=+26.691839219" watchObservedRunningTime="2026-07-23 00:23:52.896467539 +0000 UTC m=+26.692346812"
Jul 23 00:23:54.740608 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.740364    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:54.741261 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.740369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:54.741261 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:54.740630    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:54.741261 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.740372    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:54.741261 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:54.740702    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:54.741261 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:54.740799    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:54.889209 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.889189    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:23:54.889515 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.889490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"ee5adb2cbcbeed662019a569d82f4ab6034cc230e99b9c151609d985cab091e8"}
Jul 23 00:23:54.889818 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.889802    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:54.889895 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.889856    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:54.889978 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.889958    2569 scope.go:117] "RemoveContainer" containerID="114e5eb442589c8505ec0e1094a51ef0f3521f9a09445541fdcc2a0c3c9c58a9"
Jul 23 00:23:54.905576 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.905548    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:54.905799 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:54.905782    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:55.893029 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:55.892988    2569 generic.go:356] "Generic (PLEG): container finished" podID="aee7a54d-abce-4798-9b79-bd89f42aae36" containerID="1ea1dc190f3e60061c7192af76298357ed2ee7d2d98bb0dfc4620c4f395f8d5c" exitCode=0
Jul 23 00:23:55.893562 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:55.893084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerDied","Data":"1ea1dc190f3e60061c7192af76298357ed2ee7d2d98bb0dfc4620c4f395f8d5c"}
Jul 23 00:23:55.896968 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:55.896950    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:23:55.897321 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:55.897295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" event={"ID":"280511d0-25a2-4136-8252-1b3ece74cc94","Type":"ContainerStarted","Data":"e5446645fa06e80e91099ea52ce76db48d32240e9b9f5e2d4f9234ded580fa4a"}
Jul 23 00:23:55.897432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:55.897415    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 00:23:55.937238 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:55.937178    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw" podStartSLOduration=8.63921345 podStartE2EDuration="28.937160015s" totalImagesPullingTime="20.297946565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:55.936660864 +0000 UTC m=+29.732540115" watchObservedRunningTime="2026-07-23 00:23:55.937160015 +0000 UTC m=+29.733039286"
Jul 23 00:23:56.742679 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.742499    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bvppm"]
Jul 23 00:23:56.742679 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.742609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:56.742933 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.742614    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:56.742933 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:56.742790    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:56.742933 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:56.742872    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:56.742933 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.742627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:56.743149 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:56.742952    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:56.744367 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.744346    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9dgl"]
Jul 23 00:23:56.745224 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.745206    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h74t5"]
Jul 23 00:23:56.899444 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.899415    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:56.899444 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.899446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:56.899961 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.899463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:56.899961 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:56.899592    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:56.899961 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:56.899629    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:56.899961 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:56.899692    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:56.899961 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:56.899722    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 00:23:57.903420 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:57.903389    2569 generic.go:356] "Generic (PLEG): container finished" podID="aee7a54d-abce-4798-9b79-bd89f42aae36" containerID="6c7c78fc399ddf519942cea7fa6cb519c57aaa4b3ad5f9f61664efb3d60fe0af" exitCode=0
Jul 23 00:23:57.903864 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:57.903446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerDied","Data":"6c7c78fc399ddf519942cea7fa6cb519c57aaa4b3ad5f9f61664efb3d60fe0af"}
Jul 23 00:23:58.290614 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:58.290526    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:23:58.743049 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:58.743021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:23:58.743049 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:58.743036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:23:58.743049 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:58.743048    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:23:58.743326 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:58.743150    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:23:58.743326 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:58.743262    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:23:58.743426 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:23:58.743347    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:23:59.909176 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:59.909141    2569 generic.go:356] "Generic (PLEG): container finished" podID="aee7a54d-abce-4798-9b79-bd89f42aae36" containerID="5368d8ea77865ccc9efb909a4e0686329ae08dfa42fae4f8186b6b5850e3b447" exitCode=0
Jul 23 00:23:59.909935 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:23:59.909200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerDied","Data":"5368d8ea77865ccc9efb909a4e0686329ae08dfa42fae4f8186b6b5850e3b447"}
Jul 23 00:24:00.294885 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:00.294806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:24:00.295035 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.294952    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:24:00.295035 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.295017    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:24:32.294997541 +0000 UTC m=+66.090876777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:24:00.395295 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:00.395263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:24:00.395468 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.395395    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:24:00.395468 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.395462    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret podName:6b914057-5d8f-4393-93d5-389ab25f66ad nodeName:}" failed. No retries permitted until 2026-07-23 00:24:32.395443679 +0000 UTC m=+66.191322916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret") pod "global-pull-secret-syncer-bvppm" (UID: "6b914057-5d8f-4393-93d5-389ab25f66ad") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:24:00.496538 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:00.496505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:00.496723 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.496706    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:24:00.496789 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.496732    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:24:00.496789 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.496747    2569 projected.go:196] Error preparing data for projected volume kube-api-access-mjm6h for pod openshift-network-diagnostics/network-check-target-h74t5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:24:00.496879 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.496816    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h podName:cc73ce90-e463-4ba2-ac06-c77216a2de4f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:32.496796752 +0000 UTC m=+66.292675982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mjm6h" (UniqueName: "kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h") pod "network-check-target-h74t5" (UID: "cc73ce90-e463-4ba2-ac06-c77216a2de4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:24:00.739979 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:00.739933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:24:00.740156 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:00.739943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:24:00.740156 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.740101    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bvppm" podUID="6b914057-5d8f-4393-93d5-389ab25f66ad"
Jul 23 00:24:00.740156 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:00.740132    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:00.740317 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.740229    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:24:00.740317 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:00.740298    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h74t5" podUID="cc73ce90-e463-4ba2-ac06-c77216a2de4f"
Jul 23 00:24:02.500417 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.500388    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-138.ec2.internal" event="NodeReady"
Jul 23 00:24:02.500798 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.500529    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 00:24:02.550780 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.550736    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-678wz"]
Jul 23 00:24:02.555392 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.555365    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mv4sx"]
Jul 23 00:24:02.555523 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.555464    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.558624 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.558601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.563481 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563317    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 00:24:02.563481 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563373    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vdgqd\""
Jul 23 00:24:02.563690 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563673    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 00:24:02.563749 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563714    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhdn8\""
Jul 23 00:24:02.563800 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 00:24:02.563849 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563801    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 00:24:02.564019 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.563989    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 00:24:02.572004 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.571980    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mv4sx"]
Jul 23 00:24:02.580709 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.580683    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-678wz"]
Jul 23 00:24:02.612180 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.612148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/654828e6-8725-4ac8-98c7-709acd8b391f-config-volume\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.612180 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.612183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74a905fb-0c3a-494c-a0e8-fa411db6f000-cert\") pod \"ingress-canary-mv4sx\" (UID: \"74a905fb-0c3a-494c-a0e8-fa411db6f000\") " pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.612395 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.612215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjfvd\" (UniqueName: \"kubernetes.io/projected/74a905fb-0c3a-494c-a0e8-fa411db6f000-kube-api-access-tjfvd\") pod \"ingress-canary-mv4sx\" (UID: \"74a905fb-0c3a-494c-a0e8-fa411db6f000\") " pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.612395 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.612270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.612395 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.612296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654828e6-8725-4ac8-98c7-709acd8b391f-tmp-dir\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.612395 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.612356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbtg\" (UniqueName: \"kubernetes.io/projected/654828e6-8725-4ac8-98c7-709acd8b391f-kube-api-access-fvbtg\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.713838 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.713734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbtg\" (UniqueName: \"kubernetes.io/projected/654828e6-8725-4ac8-98c7-709acd8b391f-kube-api-access-fvbtg\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.713838 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.713836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/654828e6-8725-4ac8-98c7-709acd8b391f-config-volume\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.714055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.713873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74a905fb-0c3a-494c-a0e8-fa411db6f000-cert\") pod \"ingress-canary-mv4sx\" (UID: \"74a905fb-0c3a-494c-a0e8-fa411db6f000\") " pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.714055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.713921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjfvd\" (UniqueName: \"kubernetes.io/projected/74a905fb-0c3a-494c-a0e8-fa411db6f000-kube-api-access-tjfvd\") pod \"ingress-canary-mv4sx\" (UID: \"74a905fb-0c3a-494c-a0e8-fa411db6f000\") " pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.714055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.713988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.714055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.714016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654828e6-8725-4ac8-98c7-709acd8b391f-tmp-dir\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.714438 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.714417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654828e6-8725-4ac8-98c7-709acd8b391f-tmp-dir\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.714887 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:02.714867    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:02.714958 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:02.714942    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:03.214922927 +0000 UTC m=+37.010802172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:02.716117 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.715904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/654828e6-8725-4ac8-98c7-709acd8b391f-config-volume\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.719743 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.719584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74a905fb-0c3a-494c-a0e8-fa411db6f000-cert\") pod \"ingress-canary-mv4sx\" (UID: \"74a905fb-0c3a-494c-a0e8-fa411db6f000\") " pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.723615 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.723591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbtg\" (UniqueName: \"kubernetes.io/projected/654828e6-8725-4ac8-98c7-709acd8b391f-kube-api-access-fvbtg\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:02.724388 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.724367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjfvd\" (UniqueName: \"kubernetes.io/projected/74a905fb-0c3a-494c-a0e8-fa411db6f000-kube-api-access-tjfvd\") pod \"ingress-canary-mv4sx\" (UID: \"74a905fb-0c3a-494c-a0e8-fa411db6f000\") " pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:02.743231 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.743205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:02.743362 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.743311    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:24:02.743528 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.743494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:24:02.746289 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.746239    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 00:24:02.746392 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.746301    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 00:24:02.746392 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.746350    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8gvmb\""
Jul 23 00:24:02.746520 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.746472    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c75lg\""
Jul 23 00:24:02.746846 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.746824    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 00:24:02.746940 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.746863    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 00:24:02.874521 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:02.874474    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mv4sx"
Jul 23 00:24:03.217669 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:03.217628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:03.217850 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:03.217787    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:03.217912 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:03.217855    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:04.217839552 +0000 UTC m=+38.013718785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:03.922812 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:03.922788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mv4sx"]
Jul 23 00:24:03.925878 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:24:03.925849    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a905fb_0c3a_494c_a0e8_fa411db6f000.slice/crio-02e1feeed2e1cbcf6dfeff7b4cde41c3fa9facf2ce4e5e3c4c0f26008096ce90 WatchSource:0}: Error finding container 02e1feeed2e1cbcf6dfeff7b4cde41c3fa9facf2ce4e5e3c4c0f26008096ce90: Status 404 returned error can't find the container with id 02e1feeed2e1cbcf6dfeff7b4cde41c3fa9facf2ce4e5e3c4c0f26008096ce90
Jul 23 00:24:04.225568 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:04.225533    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:04.225711 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:04.225676    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:04.225769 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:04.225741    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:06.225724973 +0000 UTC m=+40.021604202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:04.927203 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:04.927164    2569 generic.go:356] "Generic (PLEG): container finished" podID="aee7a54d-abce-4798-9b79-bd89f42aae36" containerID="e2d521d1fe876d62b7f4c9a745b7489ca5c0a76dc66386d3d2214310642fac94" exitCode=0
Jul 23 00:24:04.927663 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:04.927292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerDied","Data":"e2d521d1fe876d62b7f4c9a745b7489ca5c0a76dc66386d3d2214310642fac94"}
Jul 23 00:24:04.928945 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:04.928639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mv4sx" event={"ID":"74a905fb-0c3a-494c-a0e8-fa411db6f000","Type":"ContainerStarted","Data":"02e1feeed2e1cbcf6dfeff7b4cde41c3fa9facf2ce4e5e3c4c0f26008096ce90"}
Jul 23 00:24:05.934221 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:05.934182    2569 generic.go:356] "Generic (PLEG): container finished" podID="aee7a54d-abce-4798-9b79-bd89f42aae36" containerID="7de43bde7833d93c397fe43bb5797b535c43ac3b8d8fe55f952e47a3152b339a" exitCode=0
Jul 23 00:24:05.934723 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:05.934266    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerDied","Data":"7de43bde7833d93c397fe43bb5797b535c43ac3b8d8fe55f952e47a3152b339a"}
Jul 23 00:24:06.241246 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:06.241151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:06.241398 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:06.241317    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:06.241439 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:06.241399    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:10.241380781 +0000 UTC m=+44.037260012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:06.937546 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:06.937509    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mv4sx" event={"ID":"74a905fb-0c3a-494c-a0e8-fa411db6f000","Type":"ContainerStarted","Data":"58c32e742bea403ec207a111814c3fb338edebfe5d60584493a303edc1173779"}
Jul 23 00:24:06.940623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:06.940599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" event={"ID":"aee7a54d-abce-4798-9b79-bd89f42aae36","Type":"ContainerStarted","Data":"b2569ee0c2d89401ef7d4c185ac68e1990f12f6027b0c85c6fb8930a977fc8d3"}
Jul 23 00:24:06.952422 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:06.952381    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mv4sx" podStartSLOduration=2.420039047 podStartE2EDuration="4.952367352s" totalImagesPullingTime="2.532328305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:06.952026872 +0000 UTC m=+40.747906122" watchObservedRunningTime="2026-07-23 00:24:06.952367352 +0000 UTC m=+40.748246600"
Jul 23 00:24:06.973349 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:06.973296    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5gnx8" podStartSLOduration=9.12343172 podStartE2EDuration="39.973282207s" totalImagesPullingTime="30.849850487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:06.972263001 +0000 UTC m=+40.768142287" watchObservedRunningTime="2026-07-23 00:24:06.973282207 +0000 UTC m=+40.769161459"
Jul 23 00:24:10.271221 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:10.271180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:10.271582 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:10.271294    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:10.271582 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:10.271348    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:18.271333945 +0000 UTC m=+52.067213174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:18.329542 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:18.329500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:18.329974 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:18.329649    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:18.329974 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:18.329710    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:24:34.329692586 +0000 UTC m=+68.125571831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:28.301306 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:28.301276    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmfpw"
Jul 23 00:24:32.320636 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.320592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:24:32.323158 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.323139    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 00:24:32.331130 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:32.331110    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 00:24:32.331178 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:32.331168    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:25:36.331151351 +0000 UTC m=+130.127030583 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : secret "metrics-daemon-secret" not found
Jul 23 00:24:32.421800 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.421769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:24:32.424413 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.424397    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 00:24:32.435796 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.435774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b914057-5d8f-4393-93d5-389ab25f66ad-original-pull-secret\") pod \"global-pull-secret-syncer-bvppm\" (UID: \"6b914057-5d8f-4393-93d5-389ab25f66ad\") " pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:24:32.462119 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.462083    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bvppm"
Jul 23 00:24:32.524119 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.522986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:32.528088 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.525876    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 00:24:32.537341 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.537138    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 00:24:32.547517 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.547492    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjm6h\" (UniqueName: \"kubernetes.io/projected/cc73ce90-e463-4ba2-ac06-c77216a2de4f-kube-api-access-mjm6h\") pod \"network-check-target-h74t5\" (UID: \"cc73ce90-e463-4ba2-ac06-c77216a2de4f\") " pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:32.599341 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.599272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bvppm"]
Jul 23 00:24:32.757677 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.757649    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c75lg\""
Jul 23 00:24:32.765722 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.765702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:32.880178 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.880128    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h74t5"]
Jul 23 00:24:32.883309 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:24:32.883282    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc73ce90_e463_4ba2_ac06_c77216a2de4f.slice/crio-ef0872b823614ecd0815bc0ccc4eb133ecd4934b6972f876cb6bc8b3772b7d92 WatchSource:0}: Error finding container ef0872b823614ecd0815bc0ccc4eb133ecd4934b6972f876cb6bc8b3772b7d92: Status 404 returned error can't find the container with id ef0872b823614ecd0815bc0ccc4eb133ecd4934b6972f876cb6bc8b3772b7d92
Jul 23 00:24:32.999775 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:32.999734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bvppm" event={"ID":"6b914057-5d8f-4393-93d5-389ab25f66ad","Type":"ContainerStarted","Data":"eb8cfacf1f78671016f45f2245c3f09af1807a47dd27749192264afae4ba7947"}
Jul 23 00:24:33.000729 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:33.000705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h74t5" event={"ID":"cc73ce90-e463-4ba2-ac06-c77216a2de4f","Type":"ContainerStarted","Data":"ef0872b823614ecd0815bc0ccc4eb133ecd4934b6972f876cb6bc8b3772b7d92"}
Jul 23 00:24:34.336400 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:34.336342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:24:34.336870 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:34.336538    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:24:34.336870 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:24:34.336636    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:25:06.336611486 +0000 UTC m=+100.132490720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:24:38.014656 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:38.014618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bvppm" event={"ID":"6b914057-5d8f-4393-93d5-389ab25f66ad","Type":"ContainerStarted","Data":"274b9fb8abd5bb3b1e40af5f46a1bae3ff8eec55d9a550e7c046c60535d81aed"}
Jul 23 00:24:38.015889 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:38.015867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h74t5" event={"ID":"cc73ce90-e463-4ba2-ac06-c77216a2de4f","Type":"ContainerStarted","Data":"2d7f6b2cffd3ecb805d09ae255ec8edfc81639164aed13ac4f5c4cd8881a3af0"}
Jul 23 00:24:38.016001 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:38.015989    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:24:38.035092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:38.035035    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bvppm" podStartSLOduration=66.281586924 podStartE2EDuration="1m11.035022755s" totalImagesPullingTime="4.753435831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:38.034164211 +0000 UTC m=+71.830043503" watchObservedRunningTime="2026-07-23 00:24:38.035022755 +0000 UTC m=+71.830902005"
Jul 23 00:24:38.049351 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:24:38.049308    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h74t5" podStartSLOduration=66.582335927 podStartE2EDuration="1m11.049289289s" totalImagesPullingTime="4.466953362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:38.048863778 +0000 UTC m=+71.844743029" watchObservedRunningTime="2026-07-23 00:24:38.049289289 +0000 UTC m=+71.845168541"
Jul 23 00:25:06.358688 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:25:06.358651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:25:06.359118 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:25:06.358776    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:25:06.359118 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:25:06.358837    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:26:10.35882422 +0000 UTC m=+164.154703450 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:25:09.020945 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:25:09.020902    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h74t5"
Jul 23 00:25:36.352225 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:25:36.352185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:25:36.352694 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:25:36.352309    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 00:25:36.352694 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:25:36.352370    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:27:38.352355504 +0000 UTC m=+252.148234734 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : secret "metrics-daemon-secret" not found
Jul 23 00:26:05.568273 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:26:05.568223    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-678wz" podUID="654828e6-8725-4ac8-98c7-709acd8b391f"
Jul 23 00:26:05.766923 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:26:05.766873    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:26:06.224702 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:26:06.224675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-678wz"
Jul 23 00:26:10.377351 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:26:10.377313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:26:10.377736 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:26:10.377466    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:26:10.377736 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:26:10.377532    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls podName:654828e6-8725-4ac8-98c7-709acd8b391f nodeName:}" failed. No retries permitted until 2026-07-23 00:28:12.37751559 +0000 UTC m=+286.173394823 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls") pod "dns-default-678wz" (UID: "654828e6-8725-4ac8-98c7-709acd8b391f") : secret "dns-default-metrics-tls" not found
Jul 23 00:26:19.740272 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:26:19.740183    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:27:21.654603 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.654568    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-872t6"]
Jul 23 00:27:21.657238 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.657221    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"]
Jul 23 00:27:21.657355 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.657338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.659762 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.659737    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:27:21.659762 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.659747    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 23 00:27:21.660006 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.659978    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79b76659f8-mkbk7"]
Jul 23 00:27:21.660104 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.660091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"
Jul 23 00:27:21.660957 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.660930    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5pxfb\""
Jul 23 00:27:21.661104 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.660931    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 23 00:27:21.662552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.662528    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.667319 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667300    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 23 00:27:21.667319 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667311    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 00:27:21.667319 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667328    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 23 00:27:21.667532 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667384    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 23 00:27:21.667782 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667714    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 23 00:27:21.667782 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667736    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 00:27:21.667926 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667822    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 23 00:27:21.667926 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.667898    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-g7zg9\""
Jul 23 00:27:21.668482 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.668248    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"]
Jul 23 00:27:21.669531 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.669513    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vcbns\""
Jul 23 00:27:21.671033 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.671015    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.672876 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.672856    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 23 00:27:21.674150 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.674134    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 23 00:27:21.674227 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.674174    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wx4c4\""
Jul 23 00:27:21.674274 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.674253    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 00:27:21.674274 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.674259    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 00:27:21.674378 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.674365    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 23 00:27:21.677918 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.677900    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"]
Jul 23 00:27:21.680044 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.680024    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-872t6"]
Jul 23 00:27:21.685201 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.685184    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79b76659f8-mkbk7"]
Jul 23 00:27:21.698383 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.698364    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"]
Jul 23 00:27:21.740181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j942s\" (UniqueName: \"kubernetes.io/projected/d578e09f-5449-4d06-a9fb-a12e2247bfa9-kube-api-access-j942s\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.740181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d578e09f-5449-4d06-a9fb-a12e2247bfa9-serving-cert\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.740389 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/713f466d-e0af-4fd2-9fa2-1af6723e63b5-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.740389 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d578e09f-5449-4d06-a9fb-a12e2247bfa9-trusted-ca\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.740389 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-default-certificate\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.740389 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d578e09f-5449-4d06-a9fb-a12e2247bfa9-config\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.740389 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.740543 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740383    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsfq\" (UniqueName: \"kubernetes.io/projected/5a1dde95-970c-4e29-a9d4-8acc92d30839-kube-api-access-sgsfq\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.740543 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l78b\" (UniqueName: \"kubernetes.io/projected/3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce-kube-api-access-4l78b\") pod \"network-check-source-7bc5b6bf85-wcfrx\" (UID: \"3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"
Jul 23 00:27:21.740543 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-stats-auth\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.740543 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.740543 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740467    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2g2\" (UniqueName: \"kubernetes.io/projected/713f466d-e0af-4fd2-9fa2-1af6723e63b5-kube-api-access-zq2g2\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.740543 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.740484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.747488 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.747467    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"]
Jul 23 00:27:21.750248 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.750229    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.753934 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.753919    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fcrgs\""
Jul 23 00:27:21.754162 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.754149    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 23 00:27:21.755320 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.755280    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:27:21.755532 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.755517    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 23 00:27:21.756292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.756269    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 23 00:27:21.769868 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.769840    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"]
Jul 23 00:27:21.841462 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d578e09f-5449-4d06-a9fb-a12e2247bfa9-config\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.841462 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsfq\" (UniqueName: \"kubernetes.io/projected/5a1dde95-970c-4e29-a9d4-8acc92d30839-kube-api-access-sgsfq\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l78b\" (UniqueName: \"kubernetes.io/projected/3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce-kube-api-access-4l78b\") pod \"network-check-source-7bc5b6bf85-wcfrx\" (UID: \"3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-stats-auth\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2g2\" (UniqueName: \"kubernetes.io/projected/713f466d-e0af-4fd2-9fa2-1af6723e63b5-kube-api-access-zq2g2\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.841694 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vf9\" (UniqueName: \"kubernetes.io/projected/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-kube-api-access-n5vf9\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:21.841731    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:21.841751    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:21.841794    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls podName:713f466d-e0af-4fd2-9fa2-1af6723e63b5 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:22.341774389 +0000 UTC m=+236.137653626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8hq7b" (UID: "713f466d-e0af-4fd2-9fa2-1af6723e63b5") : secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j942s\" (UniqueName: \"kubernetes.io/projected/d578e09f-5449-4d06-a9fb-a12e2247bfa9-kube-api-access-j942s\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:21.841853    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:22.341824279 +0000 UTC m=+236.137703509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : secret "router-metrics-certs-default" not found
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:21.841876    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:22.341868442 +0000 UTC m=+236.137747670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : configmap references non-existent config key: service-ca.crt
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d578e09f-5449-4d06-a9fb-a12e2247bfa9-serving-cert\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/713f466d-e0af-4fd2-9fa2-1af6723e63b5-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.841986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.842092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.842029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d578e09f-5449-4d06-a9fb-a12e2247bfa9-trusted-ca\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.842599 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.842105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-default-certificate\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.842599 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.842582    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d578e09f-5449-4d06-a9fb-a12e2247bfa9-config\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.842712 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.842683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/713f466d-e0af-4fd2-9fa2-1af6723e63b5-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.842959 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.842831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d578e09f-5449-4d06-a9fb-a12e2247bfa9-trusted-ca\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.844433 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.844410    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d578e09f-5449-4d06-a9fb-a12e2247bfa9-serving-cert\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.844609 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.844532    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-stats-auth\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.844609 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.844553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-default-certificate\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.851268 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.851247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l78b\" (UniqueName: \"kubernetes.io/projected/3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce-kube-api-access-4l78b\") pod \"network-check-source-7bc5b6bf85-wcfrx\" (UID: \"3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"
Jul 23 00:27:21.851607 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.851586    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2g2\" (UniqueName: \"kubernetes.io/projected/713f466d-e0af-4fd2-9fa2-1af6723e63b5-kube-api-access-zq2g2\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:21.852218 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.852198    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j942s\" (UniqueName: \"kubernetes.io/projected/d578e09f-5449-4d06-a9fb-a12e2247bfa9-kube-api-access-j942s\") pod \"console-operator-565b7b569d-872t6\" (UID: \"d578e09f-5449-4d06-a9fb-a12e2247bfa9\") " pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.853191 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.853173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsfq\" (UniqueName: \"kubernetes.io/projected/5a1dde95-970c-4e29-a9d4-8acc92d30839-kube-api-access-sgsfq\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:21.943334 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.943249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.943334 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.943308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vf9\" (UniqueName: \"kubernetes.io/projected/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-kube-api-access-n5vf9\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.943529 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.943362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.943840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.943819    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.945387 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.945365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.951320 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.951301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vf9\" (UniqueName: \"kubernetes.io/projected/2d7c132e-d75d-4a98-8e1b-3209d7d998f7-kube-api-access-n5vf9\") pod \"kube-storage-version-migrator-operator-7658d9b749-2t2vd\" (UID: \"2d7c132e-d75d-4a98-8e1b-3209d7d998f7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:21.966387 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.966359    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:21.973558 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:21.973537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"
Jul 23 00:27:22.058689 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.058643    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"
Jul 23 00:27:22.100028 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.100000    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-872t6"]
Jul 23 00:27:22.104027 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:22.103994    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd578e09f_5449_4d06_a9fb_a12e2247bfa9.slice/crio-c3fcd554b8d7b00d463979995fac0b7687632ece6b480ba0bc31c2b8a05cee00 WatchSource:0}: Error finding container c3fcd554b8d7b00d463979995fac0b7687632ece6b480ba0bc31c2b8a05cee00: Status 404 returned error can't find the container with id c3fcd554b8d7b00d463979995fac0b7687632ece6b480ba0bc31c2b8a05cee00
Jul 23 00:27:22.120705 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.120677    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx"]
Jul 23 00:27:22.123810 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:22.123785    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9bc2a1_1b4a_44b9_8781_32f55c0ee6ce.slice/crio-fa366a3b51aa7cc541df02cbbb6bac194b7551bb7b345cb8829d464212434c96 WatchSource:0}: Error finding container fa366a3b51aa7cc541df02cbbb6bac194b7551bb7b345cb8829d464212434c96: Status 404 returned error can't find the container with id fa366a3b51aa7cc541df02cbbb6bac194b7551bb7b345cb8829d464212434c96
Jul 23 00:27:22.187280 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.187252    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd"]
Jul 23 00:27:22.191754 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:22.191728    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7c132e_d75d_4a98_8e1b_3209d7d998f7.slice/crio-3be6decf891f6a69c70d926f7d1c20499cb51a5da8f249b03b27277298571bd7 WatchSource:0}: Error finding container 3be6decf891f6a69c70d926f7d1c20499cb51a5da8f249b03b27277298571bd7: Status 404 returned error can't find the container with id 3be6decf891f6a69c70d926f7d1c20499cb51a5da8f249b03b27277298571bd7
Jul 23 00:27:22.347743 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.347711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:22.347909 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.347757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:22.347909 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:22.347881    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:23.347862162 +0000 UTC m=+237.143741392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : configmap references non-existent config key: service-ca.crt
Jul 23 00:27:22.347990 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:22.347930    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:22.347990 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:22.347973    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls podName:713f466d-e0af-4fd2-9fa2-1af6723e63b5 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:23.347958941 +0000 UTC m=+237.143838170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8hq7b" (UID: "713f466d-e0af-4fd2-9fa2-1af6723e63b5") : secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:22.348058 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.348004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:22.348140 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:22.348128    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 00:27:22.348179 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:22.348170    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:23.348158293 +0000 UTC m=+237.144037531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : secret "router-metrics-certs-default" not found
Jul 23 00:27:22.399065 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.399039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd" event={"ID":"2d7c132e-d75d-4a98-8e1b-3209d7d998f7","Type":"ContainerStarted","Data":"3be6decf891f6a69c70d926f7d1c20499cb51a5da8f249b03b27277298571bd7"}
Jul 23 00:27:22.400400 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.400377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx" event={"ID":"3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce","Type":"ContainerStarted","Data":"ddb488f6bf943b059cfbc7d89c2d4efe1989289e9729c8a02728df188aa40b47"}
Jul 23 00:27:22.400515 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.400412    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx" event={"ID":"3b9bc2a1-1b4a-44b9-8781-32f55c0ee6ce","Type":"ContainerStarted","Data":"fa366a3b51aa7cc541df02cbbb6bac194b7551bb7b345cb8829d464212434c96"}
Jul 23 00:27:22.401473 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.401454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-872t6" event={"ID":"d578e09f-5449-4d06-a9fb-a12e2247bfa9","Type":"ContainerStarted","Data":"c3fcd554b8d7b00d463979995fac0b7687632ece6b480ba0bc31c2b8a05cee00"}
Jul 23 00:27:22.418105 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:22.418038    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wcfrx" podStartSLOduration=1.418020558 podStartE2EDuration="1.418020558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:22.417751884 +0000 UTC m=+236.213631145" watchObservedRunningTime="2026-07-23 00:27:22.418020558 +0000 UTC m=+236.213899812"
Jul 23 00:27:23.356247 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:23.356200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:23.356268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:23.356301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:23.356375    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:25.356355501 +0000 UTC m=+239.152234754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : configmap references non-existent config key: service-ca.crt
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:23.356440    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:23.356481    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:25.356469256 +0000 UTC m=+239.152348490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : secret "router-metrics-certs-default" not found
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:23.356546    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:23.356654 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:23.356613    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls podName:713f466d-e0af-4fd2-9fa2-1af6723e63b5 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:25.356595019 +0000 UTC m=+239.152474250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8hq7b" (UID: "713f466d-e0af-4fd2-9fa2-1af6723e63b5") : secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:25.372177 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.372143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:25.372494 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.372184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:25.372494 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.372279    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 00:27:25.372494 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.372298    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:25.372494 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.372335    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.372321378 +0000 UTC m=+243.168200607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : secret "router-metrics-certs-default" not found
Jul 23 00:27:25.372494 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.372364    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls podName:713f466d-e0af-4fd2-9fa2-1af6723e63b5 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.372345902 +0000 UTC m=+243.168225138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8hq7b" (UID: "713f466d-e0af-4fd2-9fa2-1af6723e63b5") : secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:25.372494 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.372416    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:25.372680 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.372512    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.372501604 +0000 UTC m=+243.168380838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : configmap references non-existent config key: service-ca.crt
Jul 23 00:27:25.416033 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.415958    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd" event={"ID":"2d7c132e-d75d-4a98-8e1b-3209d7d998f7","Type":"ContainerStarted","Data":"6b7312b2d40b4814bbad5fa77337b36a45c9470ca5e4edf78ea276b530feac31"}
Jul 23 00:27:25.417615 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.417591    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/0.log"
Jul 23 00:27:25.417724 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.417627    2569 generic.go:356] "Generic (PLEG): container finished" podID="d578e09f-5449-4d06-a9fb-a12e2247bfa9" containerID="8355bd6874a12970d7311369ddb05a3d145a8969afd4cb53c69ff2cc39dc8dae" exitCode=255
Jul 23 00:27:25.417724 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.417677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-872t6" event={"ID":"d578e09f-5449-4d06-a9fb-a12e2247bfa9","Type":"ContainerDied","Data":"8355bd6874a12970d7311369ddb05a3d145a8969afd4cb53c69ff2cc39dc8dae"}
Jul 23 00:27:25.417856 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.417843    2569 scope.go:117] "RemoveContainer" containerID="8355bd6874a12970d7311369ddb05a3d145a8969afd4cb53c69ff2cc39dc8dae"
Jul 23 00:27:25.432766 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.432721    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd" podStartSLOduration=1.484662538 podStartE2EDuration="4.432709441s" totalImagesPullingTime="2.948046903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:25.431857028 +0000 UTC m=+239.227736280" watchObservedRunningTime="2026-07-23 00:27:25.432709441 +0000 UTC m=+239.228588692"
Jul 23 00:27:25.592842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.592810    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4wzgt"]
Jul 23 00:27:25.596000 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.595983    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.598263 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.598232    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 00:27:25.598536 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.598516    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 00:27:25.598602 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.598519    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fm2gr\""
Jul 23 00:27:25.598656 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.598605    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 00:27:25.598656 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.598632    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 00:27:25.610867 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.610841    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wzgt"]
Jul 23 00:27:25.675604 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.675547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d92b989-6efc-4762-8189-133049200055-crio-socket\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.675604 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.675585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d92b989-6efc-4762-8189-133049200055-data-volume\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.675604 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.675603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.675807 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.675642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwp7\" (UniqueName: \"kubernetes.io/projected/0d92b989-6efc-4762-8189-133049200055-kube-api-access-jkwp7\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.675807 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.675704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776022    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d92b989-6efc-4762-8189-133049200055-crio-socket\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776206 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d92b989-6efc-4762-8189-133049200055-data-volume\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776206 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776206 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwp7\" (UniqueName: \"kubernetes.io/projected/0d92b989-6efc-4762-8189-133049200055-kube-api-access-jkwp7\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776206 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776206 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d92b989-6efc-4762-8189-133049200055-crio-socket\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.776412 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.776248    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:25.776412 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.776298    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:27:25.776412 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.776309    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:26.276291342 +0000 UTC m=+240.072170589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:25.776522 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:25.776454    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:26.276438588 +0000 UTC m=+240.072317824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : configmap "kube-rbac-proxy" not found
Jul 23 00:27:25.776522 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.776450    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d92b989-6efc-4762-8189-133049200055-data-volume\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:25.783693 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:25.783672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwp7\" (UniqueName: \"kubernetes.io/projected/0d92b989-6efc-4762-8189-133049200055-kube-api-access-jkwp7\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:26.280279 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.280227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:26.280467 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.280382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:26.280543 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:26.280517    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:26.280595 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:26.280576    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:27.280558258 +0000 UTC m=+241.076437488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:26.280846 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:26.280743    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:27:26.280846 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:26.280824    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:27.280806726 +0000 UTC m=+241.076685973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : configmap "kube-rbac-proxy" not found
Jul 23 00:27:26.424246 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.424220    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/1.log"
Jul 23 00:27:26.424666 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.424649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/0.log"
Jul 23 00:27:26.424712 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.424691    2569 generic.go:356] "Generic (PLEG): container finished" podID="d578e09f-5449-4d06-a9fb-a12e2247bfa9" containerID="3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26" exitCode=255
Jul 23 00:27:26.424811 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.424778    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-872t6" event={"ID":"d578e09f-5449-4d06-a9fb-a12e2247bfa9","Type":"ContainerDied","Data":"3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26"}
Jul 23 00:27:26.425443 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.425423    2569 scope.go:117] "RemoveContainer" containerID="8355bd6874a12970d7311369ddb05a3d145a8969afd4cb53c69ff2cc39dc8dae"
Jul 23 00:27:26.425552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:26.425528    2569 scope.go:117] "RemoveContainer" containerID="3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26"
Jul 23 00:27:26.425721 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:26.425688    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-872t6_openshift-console-operator(d578e09f-5449-4d06-a9fb-a12e2247bfa9)\"" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podUID="d578e09f-5449-4d06-a9fb-a12e2247bfa9"
Jul 23 00:27:27.287474 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:27.287418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:27.287752 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:27.287487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:27.287752 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:27.287562    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:27.287752 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:27.287621    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:27:27.287752 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:27.287631    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.287615753 +0000 UTC m=+243.083494983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:27.287752 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:27.287689    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.287676459 +0000 UTC m=+243.083555691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : configmap "kube-rbac-proxy" not found
Jul 23 00:27:27.428877 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:27.428852    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/1.log"
Jul 23 00:27:27.429257 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:27.429180    2569 scope.go:117] "RemoveContainer" containerID="3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26"
Jul 23 00:27:27.429353 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:27.429335    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-872t6_openshift-console-operator(d578e09f-5449-4d06-a9fb-a12e2247bfa9)\"" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podUID="d578e09f-5449-4d06-a9fb-a12e2247bfa9"
Jul 23 00:27:27.664824 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:27.664801    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dl9s5_2f49d87a-dd38-40f7-9d59-526873d06917/dns-node-resolver/0.log"
Jul 23 00:27:28.464898 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:28.464870    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6tccw_aa1702e8-c63c-4224-b143-454913151172/node-ca/0.log"
Jul 23 00:27:29.305188 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:29.305151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:29.305366 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:29.305198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:29.305366 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.305278    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:27:29.305366 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.305332    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:33.305317319 +0000 UTC m=+247.101196549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : configmap "kube-rbac-proxy" not found
Jul 23 00:27:29.305477 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.305410    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:29.305477 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.305461    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:33.305449925 +0000 UTC m=+247.101329170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:29.405719 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:29.405689    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:29.405847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:29.405749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:29.405847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:29.405771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:29.405847 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.405838    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:29.405939 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.405880    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 23 00:27:29.405939 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.405909    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls podName:713f466d-e0af-4fd2-9fa2-1af6723e63b5 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:37.405891354 +0000 UTC m=+251.201770606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8hq7b" (UID: "713f466d-e0af-4fd2-9fa2-1af6723e63b5") : secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:29.405939 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.405929    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:37.405917645 +0000 UTC m=+251.201796874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : configmap references non-existent config key: service-ca.crt
Jul 23 00:27:29.406038 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:29.405941    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:37.405936009 +0000 UTC m=+251.201815237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : secret "router-metrics-certs-default" not found
Jul 23 00:27:29.464376 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:29.464355    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mv4sx_74a905fb-0c3a-494c-a0e8-fa411db6f000/serve-healthcheck-canary/0.log"
Jul 23 00:27:31.966990 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:31.966910    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:31.966990 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:31.966966    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:31.967378 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:31.967288    2569 scope.go:117] "RemoveContainer" containerID="3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26"
Jul 23 00:27:31.967450 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:31.967431    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-872t6_openshift-console-operator(d578e09f-5449-4d06-a9fb-a12e2247bfa9)\"" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podUID="d578e09f-5449-4d06-a9fb-a12e2247bfa9"
Jul 23 00:27:33.338445 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:33.338412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:33.338813 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:33.338453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:33.338813 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:33.338558    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:27:33.338813 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:33.338612    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:41.338598849 +0000 UTC m=+255.134478077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : configmap "kube-rbac-proxy" not found
Jul 23 00:27:33.338813 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:33.338558    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:33.338813 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:33.338693    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:41.338680895 +0000 UTC m=+255.134560124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:37.472429 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:37.472392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:37.472842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:37.472479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:37.472842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:37.472519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:37.472842 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:37.472555    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle podName:5a1dde95-970c-4e29-a9d4-8acc92d30839 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:53.472537391 +0000 UTC m=+267.268416621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle") pod "router-default-79b76659f8-mkbk7" (UID: "5a1dde95-970c-4e29-a9d4-8acc92d30839") : configmap references non-existent config key: service-ca.crt
Jul 23 00:27:37.472842 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:37.472611    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:37.472842 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:37.472674    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls podName:713f466d-e0af-4fd2-9fa2-1af6723e63b5 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:53.472661736 +0000 UTC m=+267.268540966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8hq7b" (UID: "713f466d-e0af-4fd2-9fa2-1af6723e63b5") : secret "cluster-monitoring-operator-tls" not found
Jul 23 00:27:37.474839 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:37.474820    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a1dde95-970c-4e29-a9d4-8acc92d30839-metrics-certs\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:38.380832 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:38.380794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:27:38.381004 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:38.380920    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 00:27:38.381004 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:38.380981    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs podName:d0d5d50e-1891-4bba-bf96-30fff80caffd nodeName:}" failed. No retries permitted until 2026-07-23 00:29:40.380967747 +0000 UTC m=+374.176846977 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs") pod "network-metrics-daemon-p9dgl" (UID: "d0d5d50e-1891-4bba-bf96-30fff80caffd") : secret "metrics-daemon-secret" not found
Jul 23 00:27:41.402805 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:41.402765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:41.402805 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:41.402809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:41.403240 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:41.402922    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:41.403240 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:41.402987    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls podName:0d92b989-6efc-4762-8189-133049200055 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:57.402972676 +0000 UTC m=+271.198851905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4wzgt" (UID: "0d92b989-6efc-4762-8189-133049200055") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:27:41.403313 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:41.403265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d92b989-6efc-4762-8189-133049200055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:46.741398 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:46.741362    2569 scope.go:117] "RemoveContainer" containerID="3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26"
Jul 23 00:27:47.485085 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:47.485049    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:27:47.485477 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:47.485463    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/1.log"
Jul 23 00:27:47.485524 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:47.485496    2569 generic.go:356] "Generic (PLEG): container finished" podID="d578e09f-5449-4d06-a9fb-a12e2247bfa9" containerID="ebbf739f31f48b5639fafb9290319144a08b2328be3e27c8e089f41a36339b19" exitCode=255
Jul 23 00:27:47.485562 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:47.485522    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-872t6" event={"ID":"d578e09f-5449-4d06-a9fb-a12e2247bfa9","Type":"ContainerDied","Data":"ebbf739f31f48b5639fafb9290319144a08b2328be3e27c8e089f41a36339b19"}
Jul 23 00:27:47.485594 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:47.485566    2569 scope.go:117] "RemoveContainer" containerID="3e10cd9856ddbf2d3b574f642409ba09d101e5d39d2cf48e64b3582872752f26"
Jul 23 00:27:47.485842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:47.485827    2569 scope.go:117] "RemoveContainer" containerID="ebbf739f31f48b5639fafb9290319144a08b2328be3e27c8e089f41a36339b19"
Jul 23 00:27:47.486005 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:47.485983    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-872t6_openshift-console-operator(d578e09f-5449-4d06-a9fb-a12e2247bfa9)\"" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podUID="d578e09f-5449-4d06-a9fb-a12e2247bfa9"
Jul 23 00:27:48.489362 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:48.489333    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:27:51.804162 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.804123    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-797b774bfc-2dfzh"]
Jul 23 00:27:51.806978 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.806960    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.809300 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.809281    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 00:27:51.810308 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.810290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 00:27:51.810360 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.810290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6v7ht\""
Jul 23 00:27:51.810413 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.810315    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 00:27:51.815627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.815610    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 00:27:51.825151 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.825127    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797b774bfc-2dfzh"]
Jul 23 00:27:51.883680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec38751-2309-47af-877f-534965a54bdd-trusted-ca\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883801 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec38751-2309-47af-877f-534965a54bdd-installation-pull-secrets\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883801 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-bound-sa-token\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883801 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec38751-2309-47af-877f-534965a54bdd-registry-certificates\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883914 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec38751-2309-47af-877f-534965a54bdd-image-registry-private-configuration\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883914 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcb7f\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-kube-api-access-dcb7f\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883994 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec38751-2309-47af-877f-534965a54bdd-ca-trust-extracted\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.883994 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.883982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-registry-tls\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.966745 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.966716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:51.966842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.966765    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:27:51.967114 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.967099    2569 scope.go:117] "RemoveContainer" containerID="ebbf739f31f48b5639fafb9290319144a08b2328be3e27c8e089f41a36339b19"
Jul 23 00:27:51.967284 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:51.967267    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-872t6_openshift-console-operator(d578e09f-5449-4d06-a9fb-a12e2247bfa9)\"" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podUID="d578e09f-5449-4d06-a9fb-a12e2247bfa9"
Jul 23 00:27:51.984906 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.984885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec38751-2309-47af-877f-534965a54bdd-image-registry-private-configuration\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985025 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.984928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcb7f\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-kube-api-access-dcb7f\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985025 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.984965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec38751-2309-47af-877f-534965a54bdd-ca-trust-extracted\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985025 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.984998    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-registry-tls\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985211 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.985118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec38751-2309-47af-877f-534965a54bdd-trusted-ca\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985211 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.985152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec38751-2309-47af-877f-534965a54bdd-installation-pull-secrets\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985211 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.985178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-bound-sa-token\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985350 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.985253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec38751-2309-47af-877f-534965a54bdd-registry-certificates\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.985408 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.985362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec38751-2309-47af-877f-534965a54bdd-ca-trust-extracted\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.986110 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.986063    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec38751-2309-47af-877f-534965a54bdd-registry-certificates\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.986462 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.986436    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec38751-2309-47af-877f-534965a54bdd-trusted-ca\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.987748 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.987727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec38751-2309-47af-877f-534965a54bdd-image-registry-private-configuration\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.987748 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.987740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec38751-2309-47af-877f-534965a54bdd-installation-pull-secrets\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.987868 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.987773    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-registry-tls\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.994047 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.994028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcb7f\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-kube-api-access-dcb7f\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:51.994321 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:51.994302    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec38751-2309-47af-877f-534965a54bdd-bound-sa-token\") pod \"image-registry-797b774bfc-2dfzh\" (UID: \"7ec38751-2309-47af-877f-534965a54bdd\") " pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:52.116623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:52.116598    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:52.239727 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:52.239697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797b774bfc-2dfzh"]
Jul 23 00:27:52.242720 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:52.242695    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec38751_2309_47af_877f_534965a54bdd.slice/crio-a469464be61965a1bab2070018d3000b6d3d80f16cf868c0d35753457aeb4484 WatchSource:0}: Error finding container a469464be61965a1bab2070018d3000b6d3d80f16cf868c0d35753457aeb4484: Status 404 returned error can't find the container with id a469464be61965a1bab2070018d3000b6d3d80f16cf868c0d35753457aeb4484
Jul 23 00:27:52.502387 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:52.502308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh" event={"ID":"7ec38751-2309-47af-877f-534965a54bdd","Type":"ContainerStarted","Data":"8908de1c5b126c102cba4111cf64721234896006b37ac78f238cea842bdf6d88"}
Jul 23 00:27:52.502387 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:52.502365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh" event={"ID":"7ec38751-2309-47af-877f-534965a54bdd","Type":"ContainerStarted","Data":"a469464be61965a1bab2070018d3000b6d3d80f16cf868c0d35753457aeb4484"}
Jul 23 00:27:52.502604 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:52.502401    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:27:52.525288 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:52.525240    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh" podStartSLOduration=1.525225846 podStartE2EDuration="1.525225846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:52.52421713 +0000 UTC m=+266.320096393" watchObservedRunningTime="2026-07-23 00:27:52.525225846 +0000 UTC m=+266.321105097"
Jul 23 00:27:53.499364 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.499314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:53.499743 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.499427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:53.500049 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.500030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a1dde95-970c-4e29-a9d4-8acc92d30839-service-ca-bundle\") pod \"router-default-79b76659f8-mkbk7\" (UID: \"5a1dde95-970c-4e29-a9d4-8acc92d30839\") " pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:53.501737 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.501715    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/713f466d-e0af-4fd2-9fa2-1af6723e63b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8hq7b\" (UID: \"713f466d-e0af-4fd2-9fa2-1af6723e63b5\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:53.782158 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.782056    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-g7zg9\""
Jul 23 00:27:53.788262 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.788236    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wx4c4\""
Jul 23 00:27:53.790025 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.790007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:53.796406 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.796390    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"
Jul 23 00:27:53.919386 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.919283    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79b76659f8-mkbk7"]
Jul 23 00:27:53.922344 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:53.922318    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1dde95_970c_4e29_a9d4_8acc92d30839.slice/crio-5818435d2e4eefc5ad74a8b9c6d4e5c2793250fb325822191ada79d361ed3213 WatchSource:0}: Error finding container 5818435d2e4eefc5ad74a8b9c6d4e5c2793250fb325822191ada79d361ed3213: Status 404 returned error can't find the container with id 5818435d2e4eefc5ad74a8b9c6d4e5c2793250fb325822191ada79d361ed3213
Jul 23 00:27:53.931831 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:53.931807    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b"]
Jul 23 00:27:53.934142 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:53.934115    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713f466d_e0af_4fd2_9fa2_1af6723e63b5.slice/crio-441e966267994d5171751c3f61c13f696bd674c7d3807eb3571ae55c90366899 WatchSource:0}: Error finding container 441e966267994d5171751c3f61c13f696bd674c7d3807eb3571ae55c90366899: Status 404 returned error can't find the container with id 441e966267994d5171751c3f61c13f696bd674c7d3807eb3571ae55c90366899
Jul 23 00:27:54.509179 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:54.509133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b" event={"ID":"713f466d-e0af-4fd2-9fa2-1af6723e63b5","Type":"ContainerStarted","Data":"441e966267994d5171751c3f61c13f696bd674c7d3807eb3571ae55c90366899"}
Jul 23 00:27:54.510287 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:54.510265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79b76659f8-mkbk7" event={"ID":"5a1dde95-970c-4e29-a9d4-8acc92d30839","Type":"ContainerStarted","Data":"07ca0eb969dcc2c683310c63f41eeb724e2097c33571fe3d094f328f0b0e6dfa"}
Jul 23 00:27:54.510390 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:54.510304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79b76659f8-mkbk7" event={"ID":"5a1dde95-970c-4e29-a9d4-8acc92d30839","Type":"ContainerStarted","Data":"5818435d2e4eefc5ad74a8b9c6d4e5c2793250fb325822191ada79d361ed3213"}
Jul 23 00:27:54.529579 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:54.529534    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-79b76659f8-mkbk7" podStartSLOduration=33.529518932 podStartE2EDuration="33.529518932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:54.528020325 +0000 UTC m=+268.323899575" watchObservedRunningTime="2026-07-23 00:27:54.529518932 +0000 UTC m=+268.325398182"
Jul 23 00:27:54.790921 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:54.790842    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:54.793652 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:54.793626    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:55.513938 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:55.513909    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:55.515547 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:55.515524    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79b76659f8-mkbk7"
Jul 23 00:27:56.448763 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.448731    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"]
Jul 23 00:27:56.451636 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.451622    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:56.454754 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.454738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-twrm9\""
Jul 23 00:27:56.454847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.454770    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 23 00:27:56.463121 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.463100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"]
Jul 23 00:27:56.518633 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.518596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b" event={"ID":"713f466d-e0af-4fd2-9fa2-1af6723e63b5","Type":"ContainerStarted","Data":"467c06bfeb85f94a7f6ac07ec85a72b7150aa7f1afa5ae27f2419b016c27cfb1"}
Jul 23 00:27:56.525073 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.525051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c234ccd0-d9e4-4757-b723-dd902958118e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg8nb\" (UID: \"c234ccd0-d9e4-4757-b723-dd902958118e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:56.535996 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.535937    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8hq7b" podStartSLOduration=33.473472016 podStartE2EDuration="35.53592436s" totalImagesPullingTime="2.062452344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:56.535635753 +0000 UTC m=+270.331515004" watchObservedRunningTime="2026-07-23 00:27:56.53592436 +0000 UTC m=+270.331803652"
Jul 23 00:27:56.625821 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:56.625795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c234ccd0-d9e4-4757-b723-dd902958118e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg8nb\" (UID: \"c234ccd0-d9e4-4757-b723-dd902958118e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:56.625950 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:56.625921    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 23 00:27:56.626006 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:27:56.625978    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c234ccd0-d9e4-4757-b723-dd902958118e-tls-certificates podName:c234ccd0-d9e4-4757-b723-dd902958118e nodeName:}" failed. No retries permitted until 2026-07-23 00:27:57.125960719 +0000 UTC m=+270.921839948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c234ccd0-d9e4-4757-b723-dd902958118e-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-sg8nb" (UID: "c234ccd0-d9e4-4757-b723-dd902958118e") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 23 00:27:57.129357 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.129306    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c234ccd0-d9e4-4757-b723-dd902958118e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg8nb\" (UID: \"c234ccd0-d9e4-4757-b723-dd902958118e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:57.131658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.131630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c234ccd0-d9e4-4757-b723-dd902958118e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-sg8nb\" (UID: \"c234ccd0-d9e4-4757-b723-dd902958118e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:57.359829 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.359787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:57.431357 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.431327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:57.433668 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.433633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d92b989-6efc-4762-8189-133049200055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4wzgt\" (UID: \"0d92b989-6efc-4762-8189-133049200055\") " pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:57.481451 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.481422    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"]
Jul 23 00:27:57.484140 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:57.484107    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc234ccd0_d9e4_4757_b723_dd902958118e.slice/crio-37dd49d6b996aafa8ac258b317b545db4b61f2ce3ad148ab02367216a42afe15 WatchSource:0}: Error finding container 37dd49d6b996aafa8ac258b317b545db4b61f2ce3ad148ab02367216a42afe15: Status 404 returned error can't find the container with id 37dd49d6b996aafa8ac258b317b545db4b61f2ce3ad148ab02367216a42afe15
Jul 23 00:27:57.523307 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.523273    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb" event={"ID":"c234ccd0-d9e4-4757-b723-dd902958118e","Type":"ContainerStarted","Data":"37dd49d6b996aafa8ac258b317b545db4b61f2ce3ad148ab02367216a42afe15"}
Jul 23 00:27:57.707386 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.707310    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fm2gr\""
Jul 23 00:27:57.715407 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.715384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4wzgt"
Jul 23 00:27:57.839293 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:57.839260    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4wzgt"]
Jul 23 00:27:57.841832 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:27:57.841809    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d92b989_6efc_4762_8189_133049200055.slice/crio-9438370c034ec2455c5ec56ff09b26d49b6116a20e935c8a6112efe18e219f76 WatchSource:0}: Error finding container 9438370c034ec2455c5ec56ff09b26d49b6116a20e935c8a6112efe18e219f76: Status 404 returned error can't find the container with id 9438370c034ec2455c5ec56ff09b26d49b6116a20e935c8a6112efe18e219f76
Jul 23 00:27:58.528859 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:58.528760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wzgt" event={"ID":"0d92b989-6efc-4762-8189-133049200055","Type":"ContainerStarted","Data":"9438370c034ec2455c5ec56ff09b26d49b6116a20e935c8a6112efe18e219f76"}
Jul 23 00:27:59.534376 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:59.534336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb" event={"ID":"c234ccd0-d9e4-4757-b723-dd902958118e","Type":"ContainerStarted","Data":"d0e684a4b1c907d3df368ffd2146ec859d383e8071ff395542137e4eb351d8cf"}
Jul 23 00:27:59.534841 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:59.534553    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:59.539982 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:59.539958    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb"
Jul 23 00:27:59.569508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:27:59.569457    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-sg8nb" podStartSLOduration=2.138476011 podStartE2EDuration="3.56944159s" totalImagesPullingTime="1.430965579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:59.552492583 +0000 UTC m=+273.348371835" watchObservedRunningTime="2026-07-23 00:27:59.56944159 +0000 UTC m=+273.365320852"
Jul 23 00:28:00.520550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.520515    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"]
Jul 23 00:28:00.535275 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.535245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.539598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.539245    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dxthl\""
Jul 23 00:28:00.539598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.539260    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 00:28:00.539598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.539358    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 23 00:28:00.539821 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.539678    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 23 00:28:00.540802 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.540773    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"]
Jul 23 00:28:00.559609 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.559577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8faf3f05-9f00-451a-934c-2a77ea23f59a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.559727 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.559632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6rs\" (UniqueName: \"kubernetes.io/projected/8faf3f05-9f00-451a-934c-2a77ea23f59a-kube-api-access-7k6rs\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.559727 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.559666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8faf3f05-9f00-451a-934c-2a77ea23f59a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.559727 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.559713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8faf3f05-9f00-451a-934c-2a77ea23f59a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.660108 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.660060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8faf3f05-9f00-451a-934c-2a77ea23f59a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.660293 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.660141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6rs\" (UniqueName: \"kubernetes.io/projected/8faf3f05-9f00-451a-934c-2a77ea23f59a-kube-api-access-7k6rs\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.660293 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.660161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8faf3f05-9f00-451a-934c-2a77ea23f59a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.660293 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.660192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8faf3f05-9f00-451a-934c-2a77ea23f59a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.660794 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.660773    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8faf3f05-9f00-451a-934c-2a77ea23f59a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.662799 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.662775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8faf3f05-9f00-451a-934c-2a77ea23f59a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.662899 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.662873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8faf3f05-9f00-451a-934c-2a77ea23f59a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.668088 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.668054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6rs\" (UniqueName: \"kubernetes.io/projected/8faf3f05-9f00-451a-934c-2a77ea23f59a-kube-api-access-7k6rs\") pod \"prometheus-operator-5945b5cfd-p57gz\" (UID: \"8faf3f05-9f00-451a-934c-2a77ea23f59a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:00.846320 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:00.846245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"
Jul 23 00:28:02.752651 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:02.752624    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-p57gz"]
Jul 23 00:28:02.755389 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:28:02.755366    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faf3f05_9f00_451a_934c_2a77ea23f59a.slice/crio-a9499059b06235ce4241666cc3722b7f59a30a0d180ae381ab0ea33b07edc0c9 WatchSource:0}: Error finding container a9499059b06235ce4241666cc3722b7f59a30a0d180ae381ab0ea33b07edc0c9: Status 404 returned error can't find the container with id a9499059b06235ce4241666cc3722b7f59a30a0d180ae381ab0ea33b07edc0c9
Jul 23 00:28:03.551681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:03.551633    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wzgt" event={"ID":"0d92b989-6efc-4762-8189-133049200055","Type":"ContainerStarted","Data":"ea39631c951d2ddf3ddb1a15e7c7dabff7974371aca7fdac9b5a5b8ecf228635"}
Jul 23 00:28:03.553499 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:03.553469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz" event={"ID":"8faf3f05-9f00-451a-934c-2a77ea23f59a","Type":"ContainerStarted","Data":"a9499059b06235ce4241666cc3722b7f59a30a0d180ae381ab0ea33b07edc0c9"}
Jul 23 00:28:03.739889 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:03.739856    2569 scope.go:117] "RemoveContainer" containerID="ebbf739f31f48b5639fafb9290319144a08b2328be3e27c8e089f41a36339b19"
Jul 23 00:28:03.740064 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:03.740038    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-872t6_openshift-console-operator(d578e09f-5449-4d06-a9fb-a12e2247bfa9)\"" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podUID="d578e09f-5449-4d06-a9fb-a12e2247bfa9"
Jul 23 00:28:06.565179 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:06.565095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wzgt" event={"ID":"0d92b989-6efc-4762-8189-133049200055","Type":"ContainerStarted","Data":"b9d4658ea61bf240fb7cc98bbc10d925b2947000b663b84d27c841024df03dc0"}
Jul 23 00:28:06.566710 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:06.566685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz" event={"ID":"8faf3f05-9f00-451a-934c-2a77ea23f59a","Type":"ContainerStarted","Data":"341f5e83d7d683657d605e63739bf508424f2fd711764ffa9ad043e57aa552c3"}
Jul 23 00:28:06.566814 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:06.566729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz" event={"ID":"8faf3f05-9f00-451a-934c-2a77ea23f59a","Type":"ContainerStarted","Data":"4e6bbcdfd393ca4a718ec7c59a6a620df330be9412d8725984e19ea92eca1835"}
Jul 23 00:28:06.584757 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:06.584711    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-p57gz" podStartSLOduration=3.104557865 podStartE2EDuration="6.584698631s" totalImagesPullingTime="3.480140766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:06.58340081 +0000 UTC m=+280.379280061" watchObservedRunningTime="2026-07-23 00:28:06.584698631 +0000 UTC m=+280.380577881"
Jul 23 00:28:08.864367 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.864332    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v8rhl"]
Jul 23 00:28:08.878338 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.878308    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.880992 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.880953    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 00:28:08.881137 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.881110    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 00:28:08.881262 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.881224    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kwh25\""
Jul 23 00:28:08.881347 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.881326    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 00:28:08.930910 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.930884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-root\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931054 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.930923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931054 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.930944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-tls\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931054 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.931009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-sys\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931054 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.931050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-wtmp\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931283 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.931129    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-textfile\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931283 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.931187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931283 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.931225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc46e7dd-a12b-43af-9f63-2159288c8196-metrics-client-ca\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:08.931395 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:08.931297    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmscg\" (UniqueName: \"kubernetes.io/projected/fc46e7dd-a12b-43af-9f63-2159288c8196-kube-api-access-zmscg\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032254 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-root\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032428 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032428 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-tls\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032428 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-sys\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032428 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032333    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-root\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032428 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-wtmp\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032428 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-textfile\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:09.032454    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:09.032535    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-tls podName:fc46e7dd-a12b-43af-9f63-2159288c8196 nodeName:}" failed. No retries permitted until 2026-07-23 00:28:09.532512437 +0000 UTC m=+283.328391672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-tls") pod "node-exporter-v8rhl" (UID: "fc46e7dd-a12b-43af-9f63-2159288c8196") : secret "node-exporter-tls" not found
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032547    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-sys\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-wtmp\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032669    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc46e7dd-a12b-43af-9f63-2159288c8196-metrics-client-ca\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.032725 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmscg\" (UniqueName: \"kubernetes.io/projected/fc46e7dd-a12b-43af-9f63-2159288c8196-kube-api-access-zmscg\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.033065 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.032755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-textfile\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.033313 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.033293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc46e7dd-a12b-43af-9f63-2159288c8196-metrics-client-ca\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.033372 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.033295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.034806 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.034783    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.041060 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.041025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmscg\" (UniqueName: \"kubernetes.io/projected/fc46e7dd-a12b-43af-9f63-2159288c8196-kube-api-access-zmscg\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.225707 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:09.225612    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-678wz" podUID="654828e6-8725-4ac8-98c7-709acd8b391f"
Jul 23 00:28:09.538482 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.538400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-tls\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.540968 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.540929    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc46e7dd-a12b-43af-9f63-2159288c8196-node-exporter-tls\") pod \"node-exporter-v8rhl\" (UID: \"fc46e7dd-a12b-43af-9f63-2159288c8196\") " pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.580038 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.579999    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-678wz"
Jul 23 00:28:09.788912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.788841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8rhl"
Jul 23 00:28:09.858011 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:28:09.857975    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc46e7dd_a12b_43af_9f63_2159288c8196.slice/crio-11a8ce142f2637057695cc4b89737250e36871e07b537dc801ec60a576a40428 WatchSource:0}: Error finding container 11a8ce142f2637057695cc4b89737250e36871e07b537dc801ec60a576a40428: Status 404 returned error can't find the container with id 11a8ce142f2637057695cc4b89737250e36871e07b537dc801ec60a576a40428
Jul 23 00:28:09.899119 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.899095    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 00:28:09.902802 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.902780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:09.905642 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.905619    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 00:28:09.906021 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906000    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 00:28:09.906173 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906009    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8ckdn\""
Jul 23 00:28:09.906173 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906104    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 00:28:09.906414 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906400    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 00:28:09.906764 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906742    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 00:28:09.906764 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906752    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 00:28:09.906972 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906959    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 00:28:09.907100 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.906976    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 00:28:09.907163 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.907132    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 00:28:09.918901 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:09.918882    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 00:28:10.040862 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.040797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.040984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.040863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.040984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.040901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.040984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.040923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-web-config\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041127 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.040982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041127 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041127 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041127 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041127 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-out\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041310 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041141    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041310 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gbg2\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-kube-api-access-8gbg2\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041310 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-volume\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.041310 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.041242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.141740 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-volume\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.141892 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.141892 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141781    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.141892 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.141892 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:10.141897    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle podName:560f749c-2e87-4c2d-9cd4-d043d8210adb nodeName:}" failed. No retries permitted until 2026-07-23 00:28:10.641874286 +0000 UTC m=+284.437753521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb") : configmap references non-existent config key: ca-bundle.crt
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141941    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-web-config\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.141975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142006    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142122    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-out\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142472 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142472 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:10.142163    2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 23 00:28:10.142472 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbg2\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-kube-api-access-8gbg2\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.142472 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:10.142233    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls podName:560f749c-2e87-4c2d-9cd4-d043d8210adb nodeName:}" failed. No retries permitted until 2026-07-23 00:28:10.642215004 +0000 UTC m=+284.438094237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb") : secret "alertmanager-main-tls" not found
Jul 23 00:28:10.142669 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.142609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.143572 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.143231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145059    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-volume\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145224 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145305 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145467 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-out\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145666 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145650    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-web-config\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145757 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.145802 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.145786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.146469 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.146446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.150402 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.150380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbg2\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-kube-api-access-8gbg2\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.586248 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.586205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4wzgt" event={"ID":"0d92b989-6efc-4762-8189-133049200055","Type":"ContainerStarted","Data":"99ecfd6ebc1c1174392811e496843e44820e35be9c27f06d2097e5a734cc002f"}
Jul 23 00:28:10.587574 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.587543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8rhl" event={"ID":"fc46e7dd-a12b-43af-9f63-2159288c8196","Type":"ContainerStarted","Data":"11a8ce142f2637057695cc4b89737250e36871e07b537dc801ec60a576a40428"}
Jul 23 00:28:10.606126 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.606060    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4wzgt" podStartSLOduration=33.667787276 podStartE2EDuration="45.606045172s" totalImagesPullingTime="11.938257896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:27:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:10.604127492 +0000 UTC m=+284.400006744" watchObservedRunningTime="2026-07-23 00:28:10.606045172 +0000 UTC m=+284.401924423"
Jul 23 00:28:10.647087 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.647034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.647792 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.647213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.648370 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.648345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.650449 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.650418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.814454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.814428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:28:10.957135 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:10.957103    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 00:28:10.971890 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:28:10.971858    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560f749c_2e87_4c2d_9cd4_d043d8210adb.slice/crio-652d72ac60323874f8f923600e35d134b906987ea3a9a9123ebc538175dbe80f WatchSource:0}: Error finding container 652d72ac60323874f8f923600e35d134b906987ea3a9a9123ebc538175dbe80f: Status 404 returned error can't find the container with id 652d72ac60323874f8f923600e35d134b906987ea3a9a9123ebc538175dbe80f
Jul 23 00:28:11.592266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:11.592224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"652d72ac60323874f8f923600e35d134b906987ea3a9a9123ebc538175dbe80f"}
Jul 23 00:28:11.593733 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:11.593707    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc46e7dd-a12b-43af-9f63-2159288c8196" containerID="e5a4a8436e025562548c93c14c645fc5c05a0cfbd36ebc5a2653c99fb56b3ef1" exitCode=0
Jul 23 00:28:11.593859 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:11.593788    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8rhl" event={"ID":"fc46e7dd-a12b-43af-9f63-2159288c8196","Type":"ContainerDied","Data":"e5a4a8436e025562548c93c14c645fc5c05a0cfbd36ebc5a2653c99fb56b3ef1"}
Jul 23 00:28:12.121331 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.121290    2569 patch_prober.go:28] interesting pod/image-registry-797b774bfc-2dfzh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 00:28:12.121698 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.121344    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh" podUID="7ec38751-2309-47af-877f-534965a54bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 00:28:12.463756 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.463670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:28:12.465982 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.465965    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/654828e6-8725-4ac8-98c7-709acd8b391f-metrics-tls\") pod \"dns-default-678wz\" (UID: \"654828e6-8725-4ac8-98c7-709acd8b391f\") " pod="openshift-dns/dns-default-678wz"
Jul 23 00:28:12.583932 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.583903    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhdn8\""
Jul 23 00:28:12.592087 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.592057    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-678wz"
Jul 23 00:28:12.601840 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.601814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8rhl" event={"ID":"fc46e7dd-a12b-43af-9f63-2159288c8196","Type":"ContainerStarted","Data":"681045caec4e857152d3f6e9c6a83f67dee901f98fb0f05f02841ed5fef03bae"}
Jul 23 00:28:12.602053 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.601867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8rhl" event={"ID":"fc46e7dd-a12b-43af-9f63-2159288c8196","Type":"ContainerStarted","Data":"70f30e9289ba85ae75b91d592ab24ecde8c30c237f967db44846f4fb1302af1b"}
Jul 23 00:28:12.603529 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.603507    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030" exitCode=0
Jul 23 00:28:12.603644 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.603548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"}
Jul 23 00:28:12.620857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.620817    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v8rhl" podStartSLOduration=3.637474051 podStartE2EDuration="4.620802899s" totalImagesPullingTime="907.960391ms" totalInitContainerRuntime="75.368457ms" isStatefulPod=true podCreationTimestamp="2026-07-23 00:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:12.620000476 +0000 UTC m=+286.415879728" watchObservedRunningTime="2026-07-23 00:28:12.620802899 +0000 UTC m=+286.416682150"
Jul 23 00:28:12.717321 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:12.717298    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-678wz"]
Jul 23 00:28:12.719808 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:28:12.719785    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654828e6_8725_4ac8_98c7_709acd8b391f.slice/crio-7498cff25d02c307d32e8281a91037b72bdf34821c513369079c1ce377680f6b WatchSource:0}: Error finding container 7498cff25d02c307d32e8281a91037b72bdf34821c513369079c1ce377680f6b: Status 404 returned error can't find the container with id 7498cff25d02c307d32e8281a91037b72bdf34821c513369079c1ce377680f6b
Jul 23 00:28:13.509820 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.509776    2569 patch_prober.go:28] interesting pod/image-registry-797b774bfc-2dfzh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 00:28:13.510284 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.509843    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh" podUID="7ec38751-2309-47af-877f-534965a54bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 00:28:13.609783 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.609721    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-678wz" event={"ID":"654828e6-8725-4ac8-98c7-709acd8b391f","Type":"ContainerStarted","Data":"7498cff25d02c307d32e8281a91037b72bdf34821c513369079c1ce377680f6b"}
Jul 23 00:28:13.620667 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.620635    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"]
Jul 23 00:28:13.624340 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.624231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:13.626566 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.626538    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 00:28:13.626684 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.626606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sw4fh\""
Jul 23 00:28:13.632169 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.631539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"]
Jul 23 00:28:13.776747 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.776661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92d4p\" (UniqueName: \"kubernetes.io/projected/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-kube-api-access-92d4p\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:13.776747 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.776707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:13.878321 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.878283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92d4p\" (UniqueName: \"kubernetes.io/projected/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-kube-api-access-92d4p\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:13.878321 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.878332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:13.878567 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:13.878484    2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 23 00:28:13.878567 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:13.878548    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-monitoring-plugin-cert podName:e48e01a8-d3ce-4a5e-9fc2-c57221f485aa nodeName:}" failed. No retries permitted until 2026-07-23 00:28:14.378529051 +0000 UTC m=+288.174408296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-ttw4p" (UID: "e48e01a8-d3ce-4a5e-9fc2-c57221f485aa") : secret "monitoring-plugin-cert" not found
Jul 23 00:28:13.887534 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:13.887483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92d4p\" (UniqueName: \"kubernetes.io/projected/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-kube-api-access-92d4p\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:14.382442 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:14.382403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:14.385017 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:14.384990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e48e01a8-d3ce-4a5e-9fc2-c57221f485aa-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttw4p\" (UID: \"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:14.538296 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:14.538256    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:15.017513 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.016683    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:28:15.024869 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.024843    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.030150 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.030123    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 00:28:15.030150 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.030136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 00:28:15.030439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.030424    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 00:28:15.031777 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.031448    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 00:28:15.031777 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.031458    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 00:28:15.031777 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.031643    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 00:28:15.031777 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.031771    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-elp6tbjluf0oh\""
Jul 23 00:28:15.032093 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.031787    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 00:28:15.032093 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.031959    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 00:28:15.032202 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032133    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 00:28:15.032302 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032269    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 00:28:15.032491 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032474    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 00:28:15.032597 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032494    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 00:28:15.032680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032629    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wp54z\""
Jul 23 00:28:15.032680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032662    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 00:28:15.032864 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032780    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 00:28:15.032864 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.032858    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 00:28:15.049474 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.049444    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088134    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088177    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-web-config\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088267    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg85h\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-kube-api-access-lg85h\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088432 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config-out\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088610    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088761    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.088912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.089469 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.088815    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.137883 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.137668    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"]
Jul 23 00:28:15.189824 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.189787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.189975 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.189837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.189975 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.189872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.189975 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.189899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190576    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-web-config\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg85h\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-kube-api-access-lg85h\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config-out\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.190927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.191627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.191627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.190986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.191627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.191018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.191627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.191047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.191627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.191100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.191627 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.191253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.193208 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.192367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.193208 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.193027    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.195042 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.193544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.195042 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.193841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.195042 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.194315    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.195042 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.194673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.197540 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.197161    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config-out\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.198196 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.197885    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.198196 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.197985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.198196 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.198038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.198495 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.198478    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.198596 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.198544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-web-config\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.199036 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.198994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.199905 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.199861    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.199997 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.199905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.201838 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.201815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.205422 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.205405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg85h\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-kube-api-access-lg85h\") pod \"prometheus-k8s-0\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.362706 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.362678    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:15.529744 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.529711    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:28:15.531595 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:28:15.531566    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa4d1db_171f_4d62_8fe1_2ac265a6fd88.slice/crio-8860e85c15dcf8171bcdfcc69832ad5d4c49f153f5bf37593cdd3926208bdd95 WatchSource:0}: Error finding container 8860e85c15dcf8171bcdfcc69832ad5d4c49f153f5bf37593cdd3926208bdd95: Status 404 returned error can't find the container with id 8860e85c15dcf8171bcdfcc69832ad5d4c49f153f5bf37593cdd3926208bdd95
Jul 23 00:28:15.621019 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.620992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"}
Jul 23 00:28:15.621403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.621055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"}
Jul 23 00:28:15.621403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.621127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"}
Jul 23 00:28:15.621403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.621164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"}
Jul 23 00:28:15.621403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.621200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"}
Jul 23 00:28:15.622820 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.622795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-678wz" event={"ID":"654828e6-8725-4ac8-98c7-709acd8b391f","Type":"ContainerStarted","Data":"73baa7cbb6d23cbb61d1521557c8f5e10651fba4f2b7c75ccd206bf6eae3b28b"}
Jul 23 00:28:15.622927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.622842    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-678wz"
Jul 23 00:28:15.622927 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.622862    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-678wz" event={"ID":"654828e6-8725-4ac8-98c7-709acd8b391f","Type":"ContainerStarted","Data":"dd9a418fed68b2e4bea9d0662515bedefed0e99d87e578e202b8f401755162c2"}
Jul 23 00:28:15.624198 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.624170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"}
Jul 23 00:28:15.624273 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.624217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"8860e85c15dcf8171bcdfcc69832ad5d4c49f153f5bf37593cdd3926208bdd95"}
Jul 23 00:28:15.625265 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.625245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p" event={"ID":"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa","Type":"ContainerStarted","Data":"8a255a120e31151604c11e2da93037f4c8a96cb264a6ab54132c6c8a746e1dd3"}
Jul 23 00:28:15.639643 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:15.639601    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-678wz" podStartSLOduration=251.375334807 podStartE2EDuration="4m13.639586257s" totalImagesPullingTime="2.26425145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:15.639179024 +0000 UTC m=+289.435058279" watchObservedRunningTime="2026-07-23 00:28:15.639586257 +0000 UTC m=+289.435465486"
Jul 23 00:28:16.631232 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:16.631190    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb" exitCode=0
Jul 23 00:28:16.631567 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:16.631256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"}
Jul 23 00:28:17.637294 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.637165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p" event={"ID":"e48e01a8-d3ce-4a5e-9fc2-c57221f485aa","Type":"ContainerStarted","Data":"d4370908ff5237487454ee2f4d487ce07845f720840ca0f2dddcfa0f40b37be7"}
Jul 23 00:28:17.637294 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.637234    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:17.641294 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.641269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerStarted","Data":"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"}
Jul 23 00:28:17.643295 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.643272    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p"
Jul 23 00:28:17.653237 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.653195    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttw4p" podStartSLOduration=2.979554076 podStartE2EDuration="4.653182186s" totalImagesPullingTime="1.67362811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:17.652700281 +0000 UTC m=+291.448579532" watchObservedRunningTime="2026-07-23 00:28:17.653182186 +0000 UTC m=+291.449061437"
Jul 23 00:28:17.677304 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.677256    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.6800218190000002 podStartE2EDuration="8.677242246s" totalImagesPullingTime="4.981323955s" totalInitContainerRuntime="15.896472ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:17.676141832 +0000 UTC m=+291.472021084" watchObservedRunningTime="2026-07-23 00:28:17.677242246 +0000 UTC m=+291.473121496"
Jul 23 00:28:17.740372 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:17.740311    2569 scope.go:117] "RemoveContainer" containerID="ebbf739f31f48b5639fafb9290319144a08b2328be3e27c8e089f41a36339b19"
Jul 23 00:28:18.647687 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:18.647660    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:28:18.648138 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:18.647791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-872t6" event={"ID":"d578e09f-5449-4d06-a9fb-a12e2247bfa9","Type":"ContainerStarted","Data":"52059159bde298979e786810a911a3237f8347a394c33c2477952a31d18c011c"}
Jul 23 00:28:18.648552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:18.648522    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:28:18.654547 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:18.654510    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-872t6"
Jul 23 00:28:18.668919 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:18.668856    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-872t6" podStartSLOduration=54.630811444 podStartE2EDuration="57.668837043s" totalImagesPullingTime="3.038025599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:18.668116181 +0000 UTC m=+292.463995433" watchObservedRunningTime="2026-07-23 00:28:18.668837043 +0000 UTC m=+292.464716298"
Jul 23 00:28:19.656564 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:19.656526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"}
Jul 23 00:28:19.656902 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:19.656606    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"}
Jul 23 00:28:21.666995 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:21.666961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"}
Jul 23 00:28:21.667457 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:21.667023    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"}
Jul 23 00:28:21.667457 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:21.667051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"}
Jul 23 00:28:21.667457 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:21.667099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerStarted","Data":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"}
Jul 23 00:28:21.695036 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:21.694987    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.161400408 podStartE2EDuration="7.694967537s" totalImagesPullingTime="5.515023061s" totalInitContainerRuntime="18.544068ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:21.693921105 +0000 UTC m=+295.489800381" watchObservedRunningTime="2026-07-23 00:28:21.694967537 +0000 UTC m=+295.490846789"
Jul 23 00:28:22.120954 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:22.120924    2569 patch_prober.go:28] interesting pod/image-registry-797b774bfc-2dfzh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 00:28:22.121148 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:22.121014    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh" podUID="7ec38751-2309-47af-877f-534965a54bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 00:28:22.741241 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:28:22.741202    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p9dgl" podUID="d0d5d50e-1891-4bba-bf96-30fff80caffd"
Jul 23 00:28:23.509639 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:23.509607    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-797b774bfc-2dfzh"
Jul 23 00:28:25.363459 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:25.363422    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:28:25.634178 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:25.634097    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-678wz"
Jul 23 00:28:26.551733 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:26.551707    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:28:26.552194 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:26.551950    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:28:26.564247 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:26.564220    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:28:26.564389 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:26.564288    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:28:26.567963 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:26.567941    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 00:28:35.739972 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:35.739943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:28:41.742631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:41.742595    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d7c132e-d75d-4a98-8e1b-3209d7d998f7" containerID="6b7312b2d40b4814bbad5fa77337b36a45c9470ca5e4edf78ea276b530feac31" exitCode=0
Jul 23 00:28:41.742631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:41.742633    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd" event={"ID":"2d7c132e-d75d-4a98-8e1b-3209d7d998f7","Type":"ContainerDied","Data":"6b7312b2d40b4814bbad5fa77337b36a45c9470ca5e4edf78ea276b530feac31"}
Jul 23 00:28:41.743149 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:41.742904    2569 scope.go:117] "RemoveContainer" containerID="6b7312b2d40b4814bbad5fa77337b36a45c9470ca5e4edf78ea276b530feac31"
Jul 23 00:28:42.752556 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:28:42.752522    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2t2vd" event={"ID":"2d7c132e-d75d-4a98-8e1b-3209d7d998f7","Type":"ContainerStarted","Data":"c5654fb8cb053027d9f931b6d597cfd3554742ac859a993216150905d9e3be98"}
Jul 23 00:29:15.363744 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:15.363704    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:15.379311 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:15.379287    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:15.888144 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:15.888116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:29.052763 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.052686    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 00:29:29.053266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.052954    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="alertmanager" containerID="cri-o://aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb" gracePeriod=120
Jul 23 00:29:29.053266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.053021    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-metric" containerID="cri-o://c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745" gracePeriod=120
Jul 23 00:29:29.053266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.053060    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="prom-label-proxy" containerID="cri-o://9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452" gracePeriod=120
Jul 23 00:29:29.053266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.053104    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-web" containerID="cri-o://468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf" gracePeriod=120
Jul 23 00:29:29.053266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.053137    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy" containerID="cri-o://88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f" gracePeriod=120
Jul 23 00:29:29.053266 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.053224    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="config-reloader" containerID="cri-o://32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844" gracePeriod=120
Jul 23 00:29:29.929493 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929459    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452" exitCode=0
Jul 23 00:29:29.929493 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929489    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f" exitCode=0
Jul 23 00:29:29.929493 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929497    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844" exitCode=0
Jul 23 00:29:29.929493 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929504    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb" exitCode=0
Jul 23 00:29:29.929791 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929528    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"}
Jul 23 00:29:29.929791 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"}
Jul 23 00:29:29.929791 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"}
Jul 23 00:29:29.929791 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:29.929667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"}
Jul 23 00:29:30.293915 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.293892    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:29:30.354040 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354013    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-web-config\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354053    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354094    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-cluster-tls-config\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354131    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354182    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-web\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354208    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-tls-assets\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354230    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gbg2\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-kube-api-access-8gbg2\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354275    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-main-db\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354300    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354324    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-out\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354354    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354378    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-metrics-client-ca\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354485 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354401    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-volume\") pod \"560f749c-2e87-4c2d-9cd4-d043d8210adb\" (UID: \"560f749c-2e87-4c2d-9cd4-d043d8210adb\") "
Jul 23 00:29:30.354817 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.354789    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:30.357415 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.357222    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-kube-api-access-8gbg2" (OuterVolumeSpecName: "kube-api-access-8gbg2") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "kube-api-access-8gbg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:29:30.357759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.357548    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:29:30.357759 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.357602    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:29:30.357888 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.357859    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.358020 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.357986    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:30.358426 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.358388    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-out" (OuterVolumeSpecName: "config-out") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:29:30.358689 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.358653    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-volume" (OuterVolumeSpecName: "config-volume") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.358966 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.358940    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.360039 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.360012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.360268 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.360238    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.361856 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.361825    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.370210 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.370184    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-web-config" (OuterVolumeSpecName: "web-config") pod "560f749c-2e87-4c2d-9cd4-d043d8210adb" (UID: "560f749c-2e87-4c2d-9cd4-d043d8210adb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:30.455870 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455842    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-tls-assets\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.455870 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455867    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gbg2\" (UniqueName: \"kubernetes.io/projected/560f749c-2e87-4c2d-9cd4-d043d8210adb-kube-api-access-8gbg2\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.455870 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455879    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-main-db\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455888    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455897    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-out\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455906    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-main-tls\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455914    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-metrics-client-ca\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455922    2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-config-volume\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455930    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-web-config\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455938    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455947    2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-cluster-tls-config\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455957    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560f749c-2e87-4c2d-9cd4-d043d8210adb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.456063 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.455966    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/560f749c-2e87-4c2d-9cd4-d043d8210adb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:30.936433 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936398    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745" exitCode=0
Jul 23 00:29:30.936433 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936425    2569 generic.go:356] "Generic (PLEG): container finished" podID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerID="468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf" exitCode=0
Jul 23 00:29:30.936680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936496    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 00:29:30.936680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"}
Jul 23 00:29:30.936680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936670    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"}
Jul 23 00:29:30.936832 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"560f749c-2e87-4c2d-9cd4-d043d8210adb","Type":"ContainerDied","Data":"652d72ac60323874f8f923600e35d134b906987ea3a9a9123ebc538175dbe80f"}
Jul 23 00:29:30.936832 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.936730    2569 scope.go:117] "RemoveContainer" containerID="9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"
Jul 23 00:29:30.944665 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.944648    2569 scope.go:117] "RemoveContainer" containerID="c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"
Jul 23 00:29:30.951676 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.951660    2569 scope.go:117] "RemoveContainer" containerID="88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"
Jul 23 00:29:30.957494 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.957471    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 00:29:30.959227 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.959205    2569 scope.go:117] "RemoveContainer" containerID="468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"
Jul 23 00:29:30.962501 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.962477    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 00:29:30.966274 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.966259    2569 scope.go:117] "RemoveContainer" containerID="32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"
Jul 23 00:29:30.972923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.972902    2569 scope.go:117] "RemoveContainer" containerID="aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"
Jul 23 00:29:30.980461 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.980445    2569 scope.go:117] "RemoveContainer" containerID="90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"
Jul 23 00:29:30.987435 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.987416    2569 scope.go:117] "RemoveContainer" containerID="9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"
Jul 23 00:29:30.987685 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.987666    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452\": container with ID starting with 9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452 not found: ID does not exist" containerID="9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"
Jul 23 00:29:30.987739 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.987693    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"} err="failed to get container status \"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452\": rpc error: code = NotFound desc = could not find container \"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452\": container with ID starting with 9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452 not found: ID does not exist"
Jul 23 00:29:30.987739 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.987723    2569 scope.go:117] "RemoveContainer" containerID="c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"
Jul 23 00:29:30.987969 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.987950    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745\": container with ID starting with c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745 not found: ID does not exist" containerID="c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"
Jul 23 00:29:30.988018 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.987974    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"} err="failed to get container status \"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745\": rpc error: code = NotFound desc = could not find container \"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745\": container with ID starting with c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745 not found: ID does not exist"
Jul 23 00:29:30.988018 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.987992    2569 scope.go:117] "RemoveContainer" containerID="88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"
Jul 23 00:29:30.988236 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.988222    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f\": container with ID starting with 88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f not found: ID does not exist" containerID="88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"
Jul 23 00:29:30.988292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.988238    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"} err="failed to get container status \"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f\": rpc error: code = NotFound desc = could not find container \"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f\": container with ID starting with 88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f not found: ID does not exist"
Jul 23 00:29:30.988292 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.988250    2569 scope.go:117] "RemoveContainer" containerID="468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"
Jul 23 00:29:30.988498 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.988467    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf\": container with ID starting with 468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf not found: ID does not exist" containerID="468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"
Jul 23 00:29:30.988552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.988502    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"} err="failed to get container status \"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf\": rpc error: code = NotFound desc = could not find container \"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf\": container with ID starting with 468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf not found: ID does not exist"
Jul 23 00:29:30.988552 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.988521    2569 scope.go:117] "RemoveContainer" containerID="32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"
Jul 23 00:29:30.988748 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.988732    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844\": container with ID starting with 32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844 not found: ID does not exist" containerID="32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"
Jul 23 00:29:30.988784 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.988752    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"} err="failed to get container status \"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844\": rpc error: code = NotFound desc = could not find container \"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844\": container with ID starting with 32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844 not found: ID does not exist"
Jul 23 00:29:30.988784 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.988766    2569 scope.go:117] "RemoveContainer" containerID="aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"
Jul 23 00:29:30.989047 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.989022    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb\": container with ID starting with aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb not found: ID does not exist" containerID="aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"
Jul 23 00:29:30.989159 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989051    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"} err="failed to get container status \"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb\": rpc error: code = NotFound desc = could not find container \"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb\": container with ID starting with aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb not found: ID does not exist"
Jul 23 00:29:30.989159 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989081    2569 scope.go:117] "RemoveContainer" containerID="90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"
Jul 23 00:29:30.989304 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:30.989288    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030\": container with ID starting with 90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030 not found: ID does not exist" containerID="90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"
Jul 23 00:29:30.989351 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989311    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"} err="failed to get container status \"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030\": rpc error: code = NotFound desc = could not find container \"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030\": container with ID starting with 90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030 not found: ID does not exist"
Jul 23 00:29:30.989351 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989326    2569 scope.go:117] "RemoveContainer" containerID="9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"
Jul 23 00:29:30.989558 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989540    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452"} err="failed to get container status \"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452\": rpc error: code = NotFound desc = could not find container \"9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452\": container with ID starting with 9a86a9cc2a516a72e014e8176145726d311d23e182589214f1e45aeec797c452 not found: ID does not exist"
Jul 23 00:29:30.989623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989560    2569 scope.go:117] "RemoveContainer" containerID="c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"
Jul 23 00:29:30.989802 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989779    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745"} err="failed to get container status \"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745\": rpc error: code = NotFound desc = could not find container \"c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745\": container with ID starting with c3e5eead53d7692616d0ce5325797ae2bed3f2fc4ee8b7d2ba46d1cf543ac745 not found: ID does not exist"
Jul 23 00:29:30.989879 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.989804    2569 scope.go:117] "RemoveContainer" containerID="88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"
Jul 23 00:29:30.990098 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990060    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f"} err="failed to get container status \"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f\": rpc error: code = NotFound desc = could not find container \"88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f\": container with ID starting with 88b59408c93f6a6c7c484eca60cba797deac23bca94b68ac2cfa9a5345102f8f not found: ID does not exist"
Jul 23 00:29:30.990166 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990101    2569 scope.go:117] "RemoveContainer" containerID="468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"
Jul 23 00:29:30.990377 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990359    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf"} err="failed to get container status \"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf\": rpc error: code = NotFound desc = could not find container \"468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf\": container with ID starting with 468b5e6e7267925568a630ed31f5384e05016615bbcccda0a3cd9ba14ccb87cf not found: ID does not exist"
Jul 23 00:29:30.990434 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990378    2569 scope.go:117] "RemoveContainer" containerID="32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"
Jul 23 00:29:30.990593 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990574    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844"} err="failed to get container status \"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844\": rpc error: code = NotFound desc = could not find container \"32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844\": container with ID starting with 32833c9b45e94794b8f8c14541f69d2a295a82afa68634f10adda0d57aab8844 not found: ID does not exist"
Jul 23 00:29:30.990644 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990594    2569 scope.go:117] "RemoveContainer" containerID="aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"
Jul 23 00:29:30.990835 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990811    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb"} err="failed to get container status \"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb\": rpc error: code = NotFound desc = could not find container \"aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb\": container with ID starting with aa83a0ae14906d60788e2c90a3177c494320c7d6254931c3041541075be2c1eb not found: ID does not exist"
Jul 23 00:29:30.990938 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.990838    2569 scope.go:117] "RemoveContainer" containerID="90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"
Jul 23 00:29:30.991020 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:30.991004    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030"} err="failed to get container status \"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030\": rpc error: code = NotFound desc = could not find container \"90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030\": container with ID starting with 90dc0930e0ceb6967819a4631ab807ef0a6a556f6f1bbc503e5f3a9bd5001030 not found: ID does not exist"
Jul 23 00:29:32.744591 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:32.744560    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" path="/var/lib/kubelet/pods/560f749c-2e87-4c2d-9cd4-d043d8210adb/volumes"
Jul 23 00:29:33.340669 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.340634    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:29:33.340941 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.340917    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="prometheus" containerID="cri-o://7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f" gracePeriod=600
Jul 23 00:29:33.341023 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.340949    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="thanos-sidecar" containerID="cri-o://131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c" gracePeriod=600
Jul 23 00:29:33.341023 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.340969    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="config-reloader" containerID="cri-o://c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1" gracePeriod=600
Jul 23 00:29:33.341163 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.341006    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-web" containerID="cri-o://aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457" gracePeriod=600
Jul 23 00:29:33.341163 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.340950    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy" containerID="cri-o://d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545" gracePeriod=600
Jul 23 00:29:33.341163 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.341046    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989" gracePeriod=600
Jul 23 00:29:33.590509 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.590260    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:33.684697 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684669    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-trusted-ca-bundle\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.684857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684714    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-thanos-prometheus-http-client-file\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.684857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684757    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-metrics-client-certs\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.684857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684784    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config-out\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.684857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684817    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-serving-certs-ca-bundle\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.684857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-db\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684884    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-web-config\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684923    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-tls-assets\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.684950    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-kubelet-serving-ca-bundle\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685003    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg85h\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-kube-api-access-lg85h\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685029    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-tls\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685054    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-grpc-tls\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685101    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-metrics-client-ca\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685111    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.685134 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685125    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-1\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685149    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-2\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685186    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-kube-rbac-proxy\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685224    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-0\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685249    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685266    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685314    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config\") pod \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\" (UID: \"efa4d1db-171f-4d62-8fe1-2ac265a6fd88\") "
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.685550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685506    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.685984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685588    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.685984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685605    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.685984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685621    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.685984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685634    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.685984 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.685889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.687646 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.687593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config-out" (OuterVolumeSpecName: "config-out") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:29:33.687646 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.687613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:29:33.687833 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.687657    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.687833 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.687725    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.688061 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.688038    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:29:33.688663 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.688610    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config" (OuterVolumeSpecName: "config") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.688663 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.688637    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.689393 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.689359    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.689695 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.689671    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.689990 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.689960    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:29:33.690220 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.690197    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-kube-api-access-lg85h" (OuterVolumeSpecName: "kube-api-access-lg85h") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "kube-api-access-lg85h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:29:33.690450 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.690420    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.690519 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.690439    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.690960 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.690931    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.699382 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.699362    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-web-config" (OuterVolumeSpecName: "web-config") pod "efa4d1db-171f-4d62-8fe1-2ac265a6fd88" (UID: "efa4d1db-171f-4d62-8fe1-2ac265a6fd88"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:29:33.786748 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786723    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-grpc-tls\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.786748 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786747    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-configmap-metrics-client-ca\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786757    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786766    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-kube-rbac-proxy\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786775    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786784    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786794    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786803    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786811    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786819    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-metrics-client-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786827    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-config-out\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786835    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-prometheus-k8s-db\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786843    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-web-config\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786850    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-tls-assets\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786857    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg85h\" (UniqueName: \"kubernetes.io/projected/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-kube-api-access-lg85h\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.787113 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.786865    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efa4d1db-171f-4d62-8fe1-2ac265a6fd88-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955300    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989" exitCode=0
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955329    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545" exitCode=0
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955335    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457" exitCode=0
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955341    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c" exitCode=0
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955346    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1" exitCode=0
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955351    2569 generic.go:356] "Generic (PLEG): container finished" podID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f" exitCode=0
Jul 23 00:29:33.955381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955376    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"}
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"}
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"}
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"}
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955550    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955579    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"}
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"}
Jul 23 00:29:33.955681 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.955656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efa4d1db-171f-4d62-8fe1-2ac265a6fd88","Type":"ContainerDied","Data":"8860e85c15dcf8171bcdfcc69832ad5d4c49f153f5bf37593cdd3926208bdd95"}
Jul 23 00:29:33.963336 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.963289    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:33.970228 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.970208    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:33.976672 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.976656    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:33.979349 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.979326    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:29:33.983594 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.983568    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:29:33.983657 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.983629    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:33.990026 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.990008    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:33.996948 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:33.996933    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.003622 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.003605    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.003877 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.003849    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.003943 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.003887    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"} err="failed to get container status \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist"
Jul 23 00:29:34.003943 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.003911    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.004213 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.004195    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.004269 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004219    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"} err="failed to get container status \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist"
Jul 23 00:29:34.004269 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004234    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.004442 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.004425    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.004497 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004450    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"} err="failed to get container status \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist"
Jul 23 00:29:34.004497 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004471    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.004716 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.004699    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.004788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004722    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"} err="failed to get container status \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist"
Jul 23 00:29:34.004788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004742    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.004932 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.004916    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.004989 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004937    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"} err="failed to get container status \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist"
Jul 23 00:29:34.004989 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.004954    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.005340 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.005322    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.005375 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.005345    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"} err="failed to get container status \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist"
Jul 23 00:29:34.005375 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.005360    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.005587 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:29:34.005570    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.005642 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.005594    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"} err="failed to get container status \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist"
Jul 23 00:29:34.005642 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.005612    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.005835 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.005818    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"} err="failed to get container status \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist"
Jul 23 00:29:34.005901 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.005836    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.006036 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006019    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"} err="failed to get container status \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist"
Jul 23 00:29:34.006132 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006037    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.006419 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006373    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"} err="failed to get container status \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist"
Jul 23 00:29:34.006486 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006423    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.006673 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006651    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"} err="failed to get container status \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist"
Jul 23 00:29:34.006673 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006672    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.006943 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006923    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"} err="failed to get container status \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist"
Jul 23 00:29:34.006943 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.006942    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.007222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.007201    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"} err="failed to get container status \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist"
Jul 23 00:29:34.007222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.007221    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.007547 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.007518    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"} err="failed to get container status \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist"
Jul 23 00:29:34.007547 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.007539    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.007795 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.007773    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"} err="failed to get container status \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist"
Jul 23 00:29:34.007846 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.007795    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.008046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008029    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"} err="failed to get container status \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist"
Jul 23 00:29:34.008046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008046    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.008186 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008171    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:29:34.008303 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008282    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"} err="failed to get container status \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist"
Jul 23 00:29:34.008351 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008307    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.008504 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008489    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="prom-label-proxy"
Jul 23 00:29:34.008545 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008508    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="prom-label-proxy"
Jul 23 00:29:34.008545 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008519    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="thanos-sidecar"
Jul 23 00:29:34.008545 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008528    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="thanos-sidecar"
Jul 23 00:29:34.008626 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008543    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-web"
Jul 23 00:29:34.008626 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008552    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-web"
Jul 23 00:29:34.008626 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008513    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"} err="failed to get container status \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist"
Jul 23 00:29:34.008626 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008580    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.008626 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008564    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="alertmanager"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008633    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="alertmanager"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008646    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="init-config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008655    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="init-config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008667    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008675    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008686    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008695    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008705    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-web"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008714    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-web"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008723    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="init-config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008731    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="init-config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008742    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-thanos"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008751    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-thanos"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008762    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="prometheus"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008769    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="prometheus"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008776    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008782    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="config-reloader"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008787    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008792    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy"
Jul 23 00:29:34.008788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008800    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-metric"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008806    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-metric"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008825    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"} err="failed to get container status \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008849    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008864    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="thanos-sidecar"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008873    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-metric"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-web"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008886    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008892    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="config-reloader"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008897    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy-web"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008903    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="config-reloader"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008908    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="prom-label-proxy"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008913    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="alertmanager"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008918    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="kube-rbac-proxy-thanos"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008924    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="560f749c-2e87-4c2d-9cd4-d043d8210adb" containerName="kube-rbac-proxy"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.008928    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" containerName="prometheus"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009105    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"} err="failed to get container status \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009119    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009354    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"} err="failed to get container status \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist"
Jul 23 00:29:34.009503 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009377    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.010175 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009590    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"} err="failed to get container status \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist"
Jul 23 00:29:34.010175 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009606    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.010175 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009824    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"} err="failed to get container status \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist"
Jul 23 00:29:34.010175 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.009848    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.010175 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010110    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"} err="failed to get container status \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist"
Jul 23 00:29:34.010175 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010133    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.010445 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010379    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"} err="failed to get container status \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist"
Jul 23 00:29:34.010445 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010395    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.010705 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010685    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"} err="failed to get container status \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist"
Jul 23 00:29:34.010705 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010706    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.010931 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010910    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"} err="failed to get container status \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist"
Jul 23 00:29:34.010997 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.010933    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.011598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.011322    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"} err="failed to get container status \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist"
Jul 23 00:29:34.011598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.011358    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.011598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.011589    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"} err="failed to get container status \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist"
Jul 23 00:29:34.011822 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.011609    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.012093 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.012039    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"} err="failed to get container status \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist"
Jul 23 00:29:34.012093 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.012082    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.012417 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.012381    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"} err="failed to get container status \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist"
Jul 23 00:29:34.012417 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.012412    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.012743 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.012714    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"} err="failed to get container status \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist"
Jul 23 00:29:34.012839 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.012745    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.013121 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013097    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"} err="failed to get container status \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist"
Jul 23 00:29:34.013215 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013130    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.013398 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013376    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"} err="failed to get container status \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist"
Jul 23 00:29:34.013480 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013399    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.013651 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013631    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"} err="failed to get container status \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist"
Jul 23 00:29:34.013711 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013652    2569 scope.go:117] "RemoveContainer" containerID="ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"
Jul 23 00:29:34.013864 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013849    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989"} err="failed to get container status \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": rpc error: code = NotFound desc = could not find container \"ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989\": container with ID starting with ff058c5215d1fb227b41e10ba3f46051277b00e1b376ea409e99def1f9cba989 not found: ID does not exist"
Jul 23 00:29:34.013906 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.013865    2569 scope.go:117] "RemoveContainer" containerID="d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"
Jul 23 00:29:34.014055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014040    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545"} err="failed to get container status \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": rpc error: code = NotFound desc = could not find container \"d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545\": container with ID starting with d254a5d236efdadcd01197dd80f8547773cbebc586cdf285f549187377686545 not found: ID does not exist"
Jul 23 00:29:34.014055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014054    2569 scope.go:117] "RemoveContainer" containerID="aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"
Jul 23 00:29:34.014264 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014248    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457"} err="failed to get container status \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": rpc error: code = NotFound desc = could not find container \"aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457\": container with ID starting with aded64e034ab7bde41bb1968b3a068e71d7fce392c0925e7b452911674a2e457 not found: ID does not exist"
Jul 23 00:29:34.014305 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014264    2569 scope.go:117] "RemoveContainer" containerID="131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"
Jul 23 00:29:34.014435 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014419    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c"} err="failed to get container status \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": rpc error: code = NotFound desc = could not find container \"131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c\": container with ID starting with 131f853c7ec3f78a8335cf5233b4852f74138eff1bbb0cf2ec89301752b0712c not found: ID does not exist"
Jul 23 00:29:34.014477 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014436    2569 scope.go:117] "RemoveContainer" containerID="c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"
Jul 23 00:29:34.014600 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014583    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1"} err="failed to get container status \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": rpc error: code = NotFound desc = could not find container \"c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1\": container with ID starting with c556dcd6d0a25541c2efbde4f33f60f0f663c05f16a3f91f877d9b83b00cdfc1 not found: ID does not exist"
Jul 23 00:29:34.014639 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014601    2569 scope.go:117] "RemoveContainer" containerID="7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"
Jul 23 00:29:34.014778 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014762    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f"} err="failed to get container status \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": rpc error: code = NotFound desc = could not find container \"7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f\": container with ID starting with 7b8db2d211bcc0ef1409a329781878a1326a01ff8c17525995966f023353b21f not found: ID does not exist"
Jul 23 00:29:34.014812 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014778    2569 scope.go:117] "RemoveContainer" containerID="02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"
Jul 23 00:29:34.014975 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.014960    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb"} err="failed to get container status \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": rpc error: code = NotFound desc = could not find container \"02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb\": container with ID starting with 02010d862a34d6426af2749862384eece157a54c35a255dd91b96d6479794edb not found: ID does not exist"
Jul 23 00:29:34.016356 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.016336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.018977 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.018956    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 00:29:34.018977 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.018970    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-elp6tbjluf0oh\""
Jul 23 00:29:34.019222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019022    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 00:29:34.019222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.018956    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 00:29:34.019222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019114    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 00:29:34.019222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019167    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 00:29:34.019222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019207    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 00:29:34.019530 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019361    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 00:29:34.019530 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019384    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 00:29:34.019803 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019783    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wp54z\""
Jul 23 00:29:34.019897 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019793    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 00:29:34.019897 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.019790    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 00:29:34.020235 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.020217    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 00:29:34.020290 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.020268    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 00:29:34.020550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.020476    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 00:29:34.022421 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.022402    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 00:29:34.025311 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.025289    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 00:29:34.033247 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.033224    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:29:34.089106 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089106 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089337 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089337 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089337 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089337 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a043a4fe-b025-429e-bcd0-aa66cb4646f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089337 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089337 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089475    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbjs\" (UniqueName: \"kubernetes.io/projected/a043a4fe-b025-429e-bcd0-aa66cb4646f8-kube-api-access-4tbjs\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a043a4fe-b025-429e-bcd0-aa66cb4646f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089940 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-config\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089940 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.089940 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.089736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190289 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190246    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190289 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a043a4fe-b025-429e-bcd0-aa66cb4646f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190470    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbjs\" (UniqueName: \"kubernetes.io/projected/a043a4fe-b025-429e-bcd0-aa66cb4646f8-kube-api-access-4tbjs\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a043a4fe-b025-429e-bcd0-aa66cb4646f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-config\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.190923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.191625 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.190985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.191625 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.191098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.191625 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.191139    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.191776 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.191645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.191776 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.191663    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.192873 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.192846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.193649 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.193623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.193742 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.193651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a043a4fe-b025-429e-bcd0-aa66cb4646f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.193801 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.193778    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.194157 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.194104    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.194308 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.194287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.194420 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.194397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.194827 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.194807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.195118 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.195094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a043a4fe-b025-429e-bcd0-aa66cb4646f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.195394 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.195374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.195819 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.195800    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.196001 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.195985    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.196052 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.196041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a043a4fe-b025-429e-bcd0-aa66cb4646f8-config\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.196259 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.196243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a043a4fe-b025-429e-bcd0-aa66cb4646f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.199132 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.199112    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbjs\" (UniqueName: \"kubernetes.io/projected/a043a4fe-b025-429e-bcd0-aa66cb4646f8-kube-api-access-4tbjs\") pod \"prometheus-k8s-0\" (UID: \"a043a4fe-b025-429e-bcd0-aa66cb4646f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.326508 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.326443    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:34.464193 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.464163    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 00:29:34.468791 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:29:34.468764    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda043a4fe_b025_429e_bcd0_aa66cb4646f8.slice/crio-ef8553423414ead3cab480063d25056118cb281655b68e75fc97e436fe8e43d9 WatchSource:0}: Error finding container ef8553423414ead3cab480063d25056118cb281655b68e75fc97e436fe8e43d9: Status 404 returned error can't find the container with id ef8553423414ead3cab480063d25056118cb281655b68e75fc97e436fe8e43d9
Jul 23 00:29:34.744441 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.744403    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa4d1db-171f-4d62-8fe1-2ac265a6fd88" path="/var/lib/kubelet/pods/efa4d1db-171f-4d62-8fe1-2ac265a6fd88/volumes"
Jul 23 00:29:34.961096 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.961048    2569 generic.go:356] "Generic (PLEG): container finished" podID="a043a4fe-b025-429e-bcd0-aa66cb4646f8" containerID="d4e2d36e4206b0c5e0fc0b0543ed9fdae4ce78a40978937d440aab1ffcdd4134" exitCode=0
Jul 23 00:29:34.961459 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.961128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerDied","Data":"d4e2d36e4206b0c5e0fc0b0543ed9fdae4ce78a40978937d440aab1ffcdd4134"}
Jul 23 00:29:34.961459 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:34.961192    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"ef8553423414ead3cab480063d25056118cb281655b68e75fc97e436fe8e43d9"}
Jul 23 00:29:35.968905 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.968868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"36a06e582d3032504f94270ae043fb3cdcd7f463d036a675a906c3b413a0e260"}
Jul 23 00:29:35.969403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.968957    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"4e584cd6d6536b525511abcb1c2450ca9780164bd5e594667d83aefdb6c082bb"}
Jul 23 00:29:35.969403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.968983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"ec9111793fa8d1865c977e8e3a9513f92335f161e1173808dd29863a58f0299c"}
Jul 23 00:29:35.969403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.969018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"9b166194b923e6a9c381d2e16df442e18bff86b9512eb3366378fa60af46cabb"}
Jul 23 00:29:35.969403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.969043    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"6d75d50ffbe0610923fbaeac5d32a44a3db5602be4d21f51117a47185f461ef1"}
Jul 23 00:29:35.969403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.969064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a043a4fe-b025-429e-bcd0-aa66cb4646f8","Type":"ContainerStarted","Data":"a9e6ff4528f5d92a08e703c9df47adfe69217f29629b3d9528e33642f922eeee"}
Jul 23 00:29:35.997914 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:35.997856    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9768674710000003 podStartE2EDuration="2.997839381s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.97191ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:29:35.99538506 +0000 UTC m=+369.791264329" watchObservedRunningTime="2026-07-23 00:29:35.997839381 +0000 UTC m=+369.793718633"
Jul 23 00:29:39.327253 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:39.327216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:29:40.441933 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.441892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:29:40.444126 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.444096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d5d50e-1891-4bba-bf96-30fff80caffd-metrics-certs\") pod \"network-metrics-daemon-p9dgl\" (UID: \"d0d5d50e-1891-4bba-bf96-30fff80caffd\") " pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:29:40.542967 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.542938    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8gvmb\""
Jul 23 00:29:40.550983 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.550963    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9dgl"
Jul 23 00:29:40.671291 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.671269    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9dgl"]
Jul 23 00:29:40.673114 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:29:40.673087    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d5d50e_1891_4bba_bf96_30fff80caffd.slice/crio-df5fe567540d25120b9cee06615b01c693603f2628bce491b2690870d0a9d4b1 WatchSource:0}: Error finding container df5fe567540d25120b9cee06615b01c693603f2628bce491b2690870d0a9d4b1: Status 404 returned error can't find the container with id df5fe567540d25120b9cee06615b01c693603f2628bce491b2690870d0a9d4b1
Jul 23 00:29:40.675034 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.675018    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:29:40.989047 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:40.989010    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9dgl" event={"ID":"d0d5d50e-1891-4bba-bf96-30fff80caffd","Type":"ContainerStarted","Data":"df5fe567540d25120b9cee06615b01c693603f2628bce491b2690870d0a9d4b1"}
Jul 23 00:29:41.995213 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:41.995131    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9dgl" event={"ID":"d0d5d50e-1891-4bba-bf96-30fff80caffd","Type":"ContainerStarted","Data":"f312edc4c30b435fc630d7e549600adf710da140ddd48b7635f3d6f38ff71f3d"}
Jul 23 00:29:41.995213 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:41.995180    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9dgl" event={"ID":"d0d5d50e-1891-4bba-bf96-30fff80caffd","Type":"ContainerStarted","Data":"a3ed1f2a85c30558e70264eda2a6db1b555a9bcd26f5b05ced1ffbc0ce1ff1e4"}
Jul 23 00:29:42.011903 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:29:42.011832    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p9dgl" podStartSLOduration=374.002348161 podStartE2EDuration="6m15.011813349s" totalImagesPullingTime="1.009465188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:29:42.011445023 +0000 UTC m=+375.807324274" watchObservedRunningTime="2026-07-23 00:29:42.011813349 +0000 UTC m=+375.807692603"
Jul 23 00:30:34.327622 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:30:34.327587    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:30:34.342787 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:30:34.342758    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:30:35.216614 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:30:35.216586    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 00:32:55.682298 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.682263    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"]
Jul 23 00:32:55.685526 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.685511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.689454 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.689427    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 00:32:55.689614 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.689519    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hhnpz\""
Jul 23 00:32:55.689754 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.689739    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 00:32:55.690400 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.690386    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 00:32:55.694727 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.694708    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 00:32:55.699052 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.699032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wgn\" (UniqueName: \"kubernetes.io/projected/e8407a8e-461f-48d0-977a-5fea3707098e-kube-api-access-q2wgn\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.699141 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.699062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8407a8e-461f-48d0-977a-5fea3707098e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.699141 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.699131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8407a8e-461f-48d0-977a-5fea3707098e-webhook-cert\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.699212 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.699165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8407a8e-461f-48d0-977a-5fea3707098e-manifests\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.699212 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.699191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8407a8e-461f-48d0-977a-5fea3707098e-tmp\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.731626 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.731603    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"]
Jul 23 00:32:55.799664 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.799632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wgn\" (UniqueName: \"kubernetes.io/projected/e8407a8e-461f-48d0-977a-5fea3707098e-kube-api-access-q2wgn\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.799799 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.799676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8407a8e-461f-48d0-977a-5fea3707098e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.799799 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.799721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8407a8e-461f-48d0-977a-5fea3707098e-webhook-cert\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.799799 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.799741    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8407a8e-461f-48d0-977a-5fea3707098e-manifests\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.799799 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.799766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8407a8e-461f-48d0-977a-5fea3707098e-tmp\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.800272 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.800241    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8407a8e-461f-48d0-977a-5fea3707098e-manifests\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.801943 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.801919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8407a8e-461f-48d0-977a-5fea3707098e-tmp\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.802278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.802258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8407a8e-461f-48d0-977a-5fea3707098e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.802342 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.802289    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8407a8e-461f-48d0-977a-5fea3707098e-webhook-cert\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.808283 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.808263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wgn\" (UniqueName: \"kubernetes.io/projected/e8407a8e-461f-48d0-977a-5fea3707098e-kube-api-access-q2wgn\") pod \"opendatahub-operator-controller-manager-695645577-fbzg8\" (UID: \"e8407a8e-461f-48d0-977a-5fea3707098e\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:55.995687 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:55.995597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:32:56.128398 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:56.128371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"]
Jul 23 00:32:56.130994 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:32:56.130962    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8407a8e_461f_48d0_977a_5fea3707098e.slice/crio-4e8a2237e47e1ac8479e7bd31bd6dbfa143726d7cae4595d47e737e002c9dd5a WatchSource:0}: Error finding container 4e8a2237e47e1ac8479e7bd31bd6dbfa143726d7cae4595d47e737e002c9dd5a: Status 404 returned error can't find the container with id 4e8a2237e47e1ac8479e7bd31bd6dbfa143726d7cae4595d47e737e002c9dd5a
Jul 23 00:32:56.722141 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:56.722108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8" event={"ID":"e8407a8e-461f-48d0-977a-5fea3707098e","Type":"ContainerStarted","Data":"4e8a2237e47e1ac8479e7bd31bd6dbfa143726d7cae4595d47e737e002c9dd5a"}
Jul 23 00:32:59.736114 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:32:59.736057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8" event={"ID":"e8407a8e-461f-48d0-977a-5fea3707098e","Type":"ContainerStarted","Data":"d7004ebd363b8663f084d78564bdbbba3ea0e587e5611d2354ce07e9fd33e70f"}
Jul 23 00:33:00.741129 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:00.741037    2569 generic.go:356] "Generic (PLEG): container finished" podID="e8407a8e-461f-48d0-977a-5fea3707098e" containerID="d7004ebd363b8663f084d78564bdbbba3ea0e587e5611d2354ce07e9fd33e70f" exitCode=0
Jul 23 00:33:00.744232 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:00.744209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8" event={"ID":"e8407a8e-461f-48d0-977a-5fea3707098e","Type":"ContainerDied","Data":"d7004ebd363b8663f084d78564bdbbba3ea0e587e5611d2354ce07e9fd33e70f"}
Jul 23 00:33:01.747062 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:01.747027    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8" event={"ID":"e8407a8e-461f-48d0-977a-5fea3707098e","Type":"ContainerStarted","Data":"11eed7f999fad3491776bd5f06c7e2d2b3db1654ace9a3d69c43815420beafe1"}
Jul 23 00:33:01.747466 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:01.747178    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:33:01.790313 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:01.790257    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8" podStartSLOduration=2.313062151 podStartE2EDuration="6.79024089s" totalImagesPullingTime="3.473657565s" totalInitContainerRuntime="1.003521174s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:01.788895367 +0000 UTC m=+575.584774628" watchObservedRunningTime="2026-07-23 00:33:01.79024089 +0000 UTC m=+575.586120140"
Jul 23 00:33:12.753581 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:12.753553    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-695645577-fbzg8"
Jul 23 00:33:16.122418 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.122379    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9f7m6"]
Jul 23 00:33:16.136130 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.136093    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9f7m6"]
Jul 23 00:33:16.136315 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.136156    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.138734 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.138713    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 23 00:33:16.138842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.138763    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zqkbk\""
Jul 23 00:33:16.165631 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.165605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pf77\" (UniqueName: \"kubernetes.io/projected/90cf7994-066e-490a-b2d3-61a861219dce-kube-api-access-9pf77\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.165739 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.165665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cf7994-066e-490a-b2d3-61a861219dce-cert\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.266619 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.266586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cf7994-066e-490a-b2d3-61a861219dce-cert\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.266796 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.266636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pf77\" (UniqueName: \"kubernetes.io/projected/90cf7994-066e-490a-b2d3-61a861219dce-kube-api-access-9pf77\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.266796 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:33:16.266761    2569 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 23 00:33:16.266879 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:33:16.266855    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cf7994-066e-490a-b2d3-61a861219dce-cert podName:90cf7994-066e-490a-b2d3-61a861219dce nodeName:}" failed. No retries permitted until 2026-07-23 00:33:16.766834255 +0000 UTC m=+590.562713498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/90cf7994-066e-490a-b2d3-61a861219dce-cert") pod "odh-model-controller-7998c67dc4-9f7m6" (UID: "90cf7994-066e-490a-b2d3-61a861219dce") : secret "odh-model-controller-webhook-cert" not found
Jul 23 00:33:16.282222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.282187    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pf77\" (UniqueName: \"kubernetes.io/projected/90cf7994-066e-490a-b2d3-61a861219dce-kube-api-access-9pf77\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.770583 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.770552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cf7994-066e-490a-b2d3-61a861219dce-cert\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:16.772903 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:16.772877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cf7994-066e-490a-b2d3-61a861219dce-cert\") pod \"odh-model-controller-7998c67dc4-9f7m6\" (UID: \"90cf7994-066e-490a-b2d3-61a861219dce\") " pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:17.046719 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.046629    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:17.199331 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.199144    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9f7m6"]
Jul 23 00:33:17.202370 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:33:17.202343    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cf7994_066e_490a_b2d3_61a861219dce.slice/crio-6a559b235e018ab5332a4c93afe7163a4849840385235ce229b2e192c8c0a96e WatchSource:0}: Error finding container 6a559b235e018ab5332a4c93afe7163a4849840385235ce229b2e192c8c0a96e: Status 404 returned error can't find the container with id 6a559b235e018ab5332a4c93afe7163a4849840385235ce229b2e192c8c0a96e
Jul 23 00:33:17.317475 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.317400    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"]
Jul 23 00:33:17.321964 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.321947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.325012 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.324979    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 00:33:17.325143 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.324979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 00:33:17.325678 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.325658    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 00:33:17.325746 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.325686    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tthvl\""
Jul 23 00:33:17.326087 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.326054    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 00:33:17.347061 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.347034    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"]
Jul 23 00:33:17.376572 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.376704 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw6lk\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-kube-api-access-rw6lk\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.376704 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376638    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09943165-d31b-4ddf-8898-9104d8563cb3-local-certs\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.376704 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376685    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-cacerts\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.376819 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.376819 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.376819 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.376795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-istio-token\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.477873 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.477834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.477891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.477936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-istio-token\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.477995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478046 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.478029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6lk\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-kube-api-access-rw6lk\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478301 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.478061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09943165-d31b-4ddf-8898-9104d8563cb3-local-certs\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478301 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.478116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-cacerts\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.478608 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.478559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.480561 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.480535    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09943165-d31b-4ddf-8898-9104d8563cb3-local-certs\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.481223 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.481198    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.481376 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.481353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-cacerts\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.481434 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.481420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.487247 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.487222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-istio-token\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.487667 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.487644    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6lk\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-kube-api-access-rw6lk\") pod \"istiod-openshift-gateway-7574459656-rx6sk\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.631333 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.631304    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:17.780697 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.780669    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"]
Jul 23 00:33:17.783690 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:33:17.783656    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09943165_d31b_4ddf_8898_9104d8563cb3.slice/crio-60b81166231b95aa1472b95aa27963fde8f96a2a69e1f096332bef6572cfda0f WatchSource:0}: Error finding container 60b81166231b95aa1472b95aa27963fde8f96a2a69e1f096332bef6572cfda0f: Status 404 returned error can't find the container with id 60b81166231b95aa1472b95aa27963fde8f96a2a69e1f096332bef6572cfda0f
Jul 23 00:33:17.810751 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.810684    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6" event={"ID":"90cf7994-066e-490a-b2d3-61a861219dce","Type":"ContainerStarted","Data":"6a559b235e018ab5332a4c93afe7163a4849840385235ce229b2e192c8c0a96e"}
Jul 23 00:33:17.812064 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:17.812042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" event={"ID":"09943165-d31b-4ddf-8898-9104d8563cb3","Type":"ContainerStarted","Data":"60b81166231b95aa1472b95aa27963fde8f96a2a69e1f096332bef6572cfda0f"}
Jul 23 00:33:21.706236 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.706193    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 23 00:33:21.706649 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.706303    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 23 00:33:21.833673 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.833583    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6" event={"ID":"90cf7994-066e-490a-b2d3-61a861219dce","Type":"ContainerStarted","Data":"13065e1ee5a555cf267bdafe5926fcaf2612ba3d9cd6c158de35c68d5e2adc5e"}
Jul 23 00:33:21.834135 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.834061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:21.835598 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.835573    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" event={"ID":"09943165-d31b-4ddf-8898-9104d8563cb3","Type":"ContainerStarted","Data":"3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1"}
Jul 23 00:33:21.835726 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.835697    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:21.852919 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.852876    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6" podStartSLOduration=1.401227351 podStartE2EDuration="5.85286167s" totalImagesPullingTime="4.451634319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:21.851311291 +0000 UTC m=+595.647190542" watchObservedRunningTime="2026-07-23 00:33:21.85286167 +0000 UTC m=+595.648740953"
Jul 23 00:33:21.880737 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:21.880677    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" podStartSLOduration=0.960475838 podStartE2EDuration="4.880657117s" totalImagesPullingTime="3.920181279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:21.879893705 +0000 UTC m=+595.675772957" watchObservedRunningTime="2026-07-23 00:33:21.880657117 +0000 UTC m=+595.676536372"
Jul 23 00:33:22.842710 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:22.842671    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rx6sk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 00:33:22.843323 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:22.843295    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" podUID="09943165-d31b-4ddf-8898-9104d8563cb3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 00:33:25.175579 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.175542    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"]
Jul 23 00:33:25.178924 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.178901    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.184903 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.184880    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:33:25.185059 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.184939    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 00:33:25.185059 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.184989    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-64hsz\""
Jul 23 00:33:25.185059 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.185052    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 00:33:25.185277 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.185138    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 00:33:25.187560 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.187542    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 00:33:25.200784 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.200760    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"]
Jul 23 00:33:25.253983 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.253960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6db51faf-0d83-4606-844e-b5589bc8475a-manager-config\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.254148 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.253999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6db51faf-0d83-4606-844e-b5589bc8475a-cert\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.254148 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.254044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh5c\" (UniqueName: \"kubernetes.io/projected/6db51faf-0d83-4606-844e-b5589bc8475a-kube-api-access-mwh5c\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.254148 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.254095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6db51faf-0d83-4606-844e-b5589bc8475a-metrics-cert\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.355269 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.355241    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6db51faf-0d83-4606-844e-b5589bc8475a-manager-config\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.355413 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.355282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6db51faf-0d83-4606-844e-b5589bc8475a-cert\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.355413 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.355316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh5c\" (UniqueName: \"kubernetes.io/projected/6db51faf-0d83-4606-844e-b5589bc8475a-kube-api-access-mwh5c\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.355413 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.355339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6db51faf-0d83-4606-844e-b5589bc8475a-metrics-cert\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.355914 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.355886    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6db51faf-0d83-4606-844e-b5589bc8475a-manager-config\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.357650 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.357622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6db51faf-0d83-4606-844e-b5589bc8475a-cert\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.357827 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.357808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6db51faf-0d83-4606-844e-b5589bc8475a-metrics-cert\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.363954 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.363932    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh5c\" (UniqueName: \"kubernetes.io/projected/6db51faf-0d83-4606-844e-b5589bc8475a-kube-api-access-mwh5c\") pod \"lws-controller-manager-85b95d447c-x2l4h\" (UID: \"6db51faf-0d83-4606-844e-b5589bc8475a\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.487827 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.487747    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:25.635721 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.635682    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"]
Jul 23 00:33:25.638479 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:33:25.638442    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db51faf_0d83_4606_844e_b5589bc8475a.slice/crio-520302a99861396d6cc0ee80480ce3dfa8a2a2a99f1fcb03fb65ba8267b6f687 WatchSource:0}: Error finding container 520302a99861396d6cc0ee80480ce3dfa8a2a2a99f1fcb03fb65ba8267b6f687: Status 404 returned error can't find the container with id 520302a99861396d6cc0ee80480ce3dfa8a2a2a99f1fcb03fb65ba8267b6f687
Jul 23 00:33:25.841581 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.841516    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:25.859412 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:25.859381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h" event={"ID":"6db51faf-0d83-4606-844e-b5589bc8475a","Type":"ContainerStarted","Data":"520302a99861396d6cc0ee80480ce3dfa8a2a2a99f1fcb03fb65ba8267b6f687"}
Jul 23 00:33:26.593084 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:26.593039    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:33:26.593541 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:26.593259    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:33:26.602575 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:26.602547    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:33:26.602758 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:26.602712    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:33:28.874933 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:28.874900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h" event={"ID":"6db51faf-0d83-4606-844e-b5589bc8475a","Type":"ContainerStarted","Data":"35e2b5df951091c20023ddff3bba4714311a8e3d4a119328849ee244c8a59687"}
Jul 23 00:33:28.875358 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:28.875035    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:33:28.891643 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:28.891599    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h" podStartSLOduration=1.442177725 podStartE2EDuration="3.891585681s" totalImagesPullingTime="2.449407956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:28.891006263 +0000 UTC m=+602.686885515" watchObservedRunningTime="2026-07-23 00:33:28.891585681 +0000 UTC m=+602.687464930"
Jul 23 00:33:29.941433 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:29.941399    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"]
Jul 23 00:33:29.941838 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:29.941595    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" podUID="09943165-d31b-4ddf-8898-9104d8563cb3" containerName="discovery" containerID="cri-o://3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1" gracePeriod=30
Jul 23 00:33:30.190013 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.189986    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:30.308763 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.308679    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-istio-token\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.308763 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.308744    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-ca-configmap\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.308939 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.308791    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw6lk\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-kube-api-access-rw6lk\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.308987 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.308941    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09943165-d31b-4ddf-8898-9104d8563cb3-local-certs\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.309054 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.309028    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-dns-cert\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.309144 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.309091    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-cacerts\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.309144 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.309123    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-kubeconfig\") pod \"09943165-d31b-4ddf-8898-9104d8563cb3\" (UID: \"09943165-d31b-4ddf-8898-9104d8563cb3\") "
Jul 23 00:33:30.309247 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.309217    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:33:30.309467 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.309433    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-ca-configmap\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.311246 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.311215    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-cacerts" (OuterVolumeSpecName: "cacerts") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:33:30.311246 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.311223    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-istio-token" (OuterVolumeSpecName: "istio-token") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:33:30.311393 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.311287    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09943165-d31b-4ddf-8898-9104d8563cb3-local-certs" (OuterVolumeSpecName: "local-certs") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:33:30.311439 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.311400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-kube-api-access-rw6lk" (OuterVolumeSpecName: "kube-api-access-rw6lk") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "kube-api-access-rw6lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:33:30.311475 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.311456    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:33:30.311595 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.311577    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "09943165-d31b-4ddf-8898-9104d8563cb3" (UID: "09943165-d31b-4ddf-8898-9104d8563cb3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:33:30.410155 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.410114    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-istio-token\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.410155 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.410150    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw6lk\" (UniqueName: \"kubernetes.io/projected/09943165-d31b-4ddf-8898-9104d8563cb3-kube-api-access-rw6lk\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.410155 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.410161    2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09943165-d31b-4ddf-8898-9104d8563cb3-local-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.410155 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.410170    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-csr-dns-cert\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.410368 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.410178    2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-cacerts\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.410368 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.410188    2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09943165-d31b-4ddf-8898-9104d8563cb3-istio-kubeconfig\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:33:30.884442 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.884411    2569 generic.go:356] "Generic (PLEG): container finished" podID="09943165-d31b-4ddf-8898-9104d8563cb3" containerID="3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1" exitCode=0
Jul 23 00:33:30.884605 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.884455    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"
Jul 23 00:33:30.884605 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.884455    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" event={"ID":"09943165-d31b-4ddf-8898-9104d8563cb3","Type":"ContainerDied","Data":"3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1"}
Jul 23 00:33:30.884605 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.884578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk" event={"ID":"09943165-d31b-4ddf-8898-9104d8563cb3","Type":"ContainerDied","Data":"60b81166231b95aa1472b95aa27963fde8f96a2a69e1f096332bef6572cfda0f"}
Jul 23 00:33:30.884605 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.884605    2569 scope.go:117] "RemoveContainer" containerID="3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1"
Jul 23 00:33:30.892719 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.892700    2569 scope.go:117] "RemoveContainer" containerID="3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1"
Jul 23 00:33:30.892979 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:33:30.892963    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1\": container with ID starting with 3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1 not found: ID does not exist" containerID="3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1"
Jul 23 00:33:30.893045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.892987    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1"} err="failed to get container status \"3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1\": rpc error: code = NotFound desc = could not find container \"3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1\": container with ID starting with 3894159449f7a68ba8c9fd20ae6ef16b3c030c476198420a04bf22c6871de5e1 not found: ID does not exist"
Jul 23 00:33:30.903286 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.903267    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"]
Jul 23 00:33:30.910270 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:30.910249    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rx6sk"]
Jul 23 00:33:32.745275 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:32.745246    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09943165-d31b-4ddf-8898-9104d8563cb3" path="/var/lib/kubelet/pods/09943165-d31b-4ddf-8898-9104d8563cb3/volumes"
Jul 23 00:33:32.842688 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:32.842661    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9f7m6"
Jul 23 00:33:39.881373 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:33:39.881344    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-x2l4h"
Jul 23 00:34:16.693809 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.693734    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44xr6"]
Jul 23 00:34:16.694281 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.694046    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09943165-d31b-4ddf-8898-9104d8563cb3" containerName="discovery"
Jul 23 00:34:16.694281 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.694057    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09943165-d31b-4ddf-8898-9104d8563cb3" containerName="discovery"
Jul 23 00:34:16.694281 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.694145    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="09943165-d31b-4ddf-8898-9104d8563cb3" containerName="discovery"
Jul 23 00:34:16.697288 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.697269    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:16.699889 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.699867    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 00:34:16.701083 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.701039    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 00:34:16.701083 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.701049    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7tnv7\""
Jul 23 00:34:16.706473 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.706450    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44xr6"]
Jul 23 00:34:16.788238 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.788208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchzx\" (UniqueName: \"kubernetes.io/projected/f067938c-2a94-4c82-a8b5-339d95dcbaa2-kube-api-access-hchzx\") pod \"kuadrant-operator-catalog-44xr6\" (UID: \"f067938c-2a94-4c82-a8b5-339d95dcbaa2\") " pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:16.888956 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.888930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchzx\" (UniqueName: \"kubernetes.io/projected/f067938c-2a94-4c82-a8b5-339d95dcbaa2-kube-api-access-hchzx\") pod \"kuadrant-operator-catalog-44xr6\" (UID: \"f067938c-2a94-4c82-a8b5-339d95dcbaa2\") " pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:16.897429 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:16.897406    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchzx\" (UniqueName: \"kubernetes.io/projected/f067938c-2a94-4c82-a8b5-339d95dcbaa2-kube-api-access-hchzx\") pod \"kuadrant-operator-catalog-44xr6\" (UID: \"f067938c-2a94-4c82-a8b5-339d95dcbaa2\") " pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:17.007945 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.007872    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:17.032901 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.032874    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44xr6"]
Jul 23 00:34:17.135751 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.135721    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44xr6"]
Jul 23 00:34:17.138718 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:34:17.138686    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf067938c_2a94_4c82_a8b5_339d95dcbaa2.slice/crio-994f15cd0d72e1219e5ae4b58f3477926b47b822fff6c58dace162210c228e3f WatchSource:0}: Error finding container 994f15cd0d72e1219e5ae4b58f3477926b47b822fff6c58dace162210c228e3f: Status 404 returned error can't find the container with id 994f15cd0d72e1219e5ae4b58f3477926b47b822fff6c58dace162210c228e3f
Jul 23 00:34:17.244135 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.244105    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7bbh2"]
Jul 23 00:34:17.250463 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.250262    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:17.255867 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.255839    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7bbh2"]
Jul 23 00:34:17.393793 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.393758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhz85\" (UniqueName: \"kubernetes.io/projected/100ba5e4-0d78-4b9c-b1e0-7b9762d87e36-kube-api-access-bhz85\") pod \"kuadrant-operator-catalog-7bbh2\" (UID: \"100ba5e4-0d78-4b9c-b1e0-7b9762d87e36\") " pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:17.494954 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.494920    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhz85\" (UniqueName: \"kubernetes.io/projected/100ba5e4-0d78-4b9c-b1e0-7b9762d87e36-kube-api-access-bhz85\") pod \"kuadrant-operator-catalog-7bbh2\" (UID: \"100ba5e4-0d78-4b9c-b1e0-7b9762d87e36\") " pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:17.503403 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.503370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhz85\" (UniqueName: \"kubernetes.io/projected/100ba5e4-0d78-4b9c-b1e0-7b9762d87e36-kube-api-access-bhz85\") pod \"kuadrant-operator-catalog-7bbh2\" (UID: \"100ba5e4-0d78-4b9c-b1e0-7b9762d87e36\") " pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:17.560946 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.560917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:17.699492 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:17.699467    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7bbh2"]
Jul 23 00:34:17.701880 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:34:17.701852    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100ba5e4_0d78_4b9c_b1e0_7b9762d87e36.slice/crio-b5552a18671e065fb340ccdf761aa6d69af30aecfbddf06a80e32b88af82dd6f WatchSource:0}: Error finding container b5552a18671e065fb340ccdf761aa6d69af30aecfbddf06a80e32b88af82dd6f: Status 404 returned error can't find the container with id b5552a18671e065fb340ccdf761aa6d69af30aecfbddf06a80e32b88af82dd6f
Jul 23 00:34:18.091257 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:18.091171    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-44xr6" event={"ID":"f067938c-2a94-4c82-a8b5-339d95dcbaa2","Type":"ContainerStarted","Data":"994f15cd0d72e1219e5ae4b58f3477926b47b822fff6c58dace162210c228e3f"}
Jul 23 00:34:18.092506 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:18.092477    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2" event={"ID":"100ba5e4-0d78-4b9c-b1e0-7b9762d87e36","Type":"ContainerStarted","Data":"b5552a18671e065fb340ccdf761aa6d69af30aecfbddf06a80e32b88af82dd6f"}
Jul 23 00:34:20.103648 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.103612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2" event={"ID":"100ba5e4-0d78-4b9c-b1e0-7b9762d87e36","Type":"ContainerStarted","Data":"001aab10942375d1c7ef134bc1574d0aed15402109ea11f5609a80c86b8d5be8"}
Jul 23 00:34:20.105038 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.104999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-44xr6" event={"ID":"f067938c-2a94-4c82-a8b5-339d95dcbaa2","Type":"ContainerStarted","Data":"47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f"}
Jul 23 00:34:20.105191 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.105093    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-44xr6" podUID="f067938c-2a94-4c82-a8b5-339d95dcbaa2" containerName="registry-server" containerID="cri-o://47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f" gracePeriod=2
Jul 23 00:34:20.120189 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.120133    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2" podStartSLOduration=1.546094213 podStartE2EDuration="3.120064357s" totalImagesPullingTime="1.573970144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:34:20.118850156 +0000 UTC m=+653.914729408" watchObservedRunningTime="2026-07-23 00:34:20.120064357 +0000 UTC m=+653.915943607"
Jul 23 00:34:20.134057 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.134018    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-44xr6" podStartSLOduration=1.9996064580000001 podStartE2EDuration="4.134005785s" totalImagesPullingTime="2.134399327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:34:20.133107191 +0000 UTC m=+653.928986441" watchObservedRunningTime="2026-07-23 00:34:20.134005785 +0000 UTC m=+653.929885038"
Jul 23 00:34:20.343803 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.343778    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:20.420373 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.420303    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hchzx\" (UniqueName: \"kubernetes.io/projected/f067938c-2a94-4c82-a8b5-339d95dcbaa2-kube-api-access-hchzx\") pod \"f067938c-2a94-4c82-a8b5-339d95dcbaa2\" (UID: \"f067938c-2a94-4c82-a8b5-339d95dcbaa2\") "
Jul 23 00:34:20.422402 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.422371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f067938c-2a94-4c82-a8b5-339d95dcbaa2-kube-api-access-hchzx" (OuterVolumeSpecName: "kube-api-access-hchzx") pod "f067938c-2a94-4c82-a8b5-339d95dcbaa2" (UID: "f067938c-2a94-4c82-a8b5-339d95dcbaa2"). InnerVolumeSpecName "kube-api-access-hchzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:34:20.521778 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:20.521748    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchzx\" (UniqueName: \"kubernetes.io/projected/f067938c-2a94-4c82-a8b5-339d95dcbaa2-kube-api-access-hchzx\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:34:21.110093 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.110038    2569 generic.go:356] "Generic (PLEG): container finished" podID="f067938c-2a94-4c82-a8b5-339d95dcbaa2" containerID="47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f" exitCode=0
Jul 23 00:34:21.110093 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.110092    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-44xr6"
Jul 23 00:34:21.110628 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.110124    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-44xr6" event={"ID":"f067938c-2a94-4c82-a8b5-339d95dcbaa2","Type":"ContainerDied","Data":"47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f"}
Jul 23 00:34:21.110628 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.110172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-44xr6" event={"ID":"f067938c-2a94-4c82-a8b5-339d95dcbaa2","Type":"ContainerDied","Data":"994f15cd0d72e1219e5ae4b58f3477926b47b822fff6c58dace162210c228e3f"}
Jul 23 00:34:21.110628 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.110193    2569 scope.go:117] "RemoveContainer" containerID="47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f"
Jul 23 00:34:21.118787 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.118772    2569 scope.go:117] "RemoveContainer" containerID="47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f"
Jul 23 00:34:21.119026 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:34:21.119009    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f\": container with ID starting with 47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f not found: ID does not exist" containerID="47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f"
Jul 23 00:34:21.119096 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.119035    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f"} err="failed to get container status \"47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f\": rpc error: code = NotFound desc = could not find container \"47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f\": container with ID starting with 47101f8cb36f07163ca4187ba7a899faa9d350712261bd53fa16ddcb9e69273f not found: ID does not exist"
Jul 23 00:34:21.126433 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.126411    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44xr6"]
Jul 23 00:34:21.127771 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:21.127751    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44xr6"]
Jul 23 00:34:22.750045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:22.749999    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f067938c-2a94-4c82-a8b5-339d95dcbaa2" path="/var/lib/kubelet/pods/f067938c-2a94-4c82-a8b5-339d95dcbaa2/volumes"
Jul 23 00:34:27.561206 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:27.561171    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:27.561613 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:27.561421    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:27.583789 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:27.583766    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:28.160395 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:28.160369    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7bbh2"
Jul 23 00:34:48.077861 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.077830    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"]
Jul 23 00:34:48.078330 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.078195    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f067938c-2a94-4c82-a8b5-339d95dcbaa2" containerName="registry-server"
Jul 23 00:34:48.078330 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.078208    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f067938c-2a94-4c82-a8b5-339d95dcbaa2" containerName="registry-server"
Jul 23 00:34:48.078330 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.078267    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f067938c-2a94-4c82-a8b5-339d95dcbaa2" containerName="registry-server"
Jul 23 00:34:48.082725 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.082709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:34:48.085390 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.085365    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-n9rck\""
Jul 23 00:34:48.093885 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.093861    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"]
Jul 23 00:34:48.136831 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.136803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6bk\" (UniqueName: \"kubernetes.io/projected/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b-kube-api-access-xm6bk\") pod \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" (UID: \"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:34:48.237754 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.237726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6bk\" (UniqueName: \"kubernetes.io/projected/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b-kube-api-access-xm6bk\") pod \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" (UID: \"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:34:48.249558 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.249522    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6bk\" (UniqueName: \"kubernetes.io/projected/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b-kube-api-access-xm6bk\") pod \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" (UID: \"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:34:48.393041 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.393006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:34:48.520416 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.520391    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"]
Jul 23 00:34:48.523045 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:34:48.523017    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8848fa06_6cc2_4e7e_bd1e_739e1d8e402b.slice/crio-c92727b818384da03193728389208287b807c0dab6356f78a3ea47491d2f8291 WatchSource:0}: Error finding container c92727b818384da03193728389208287b807c0dab6356f78a3ea47491d2f8291: Status 404 returned error can't find the container with id c92727b818384da03193728389208287b807c0dab6356f78a3ea47491d2f8291
Jul 23 00:34:48.525159 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:48.525142    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:34:49.228121 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:49.228083    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn" event={"ID":"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b","Type":"ContainerStarted","Data":"c92727b818384da03193728389208287b807c0dab6356f78a3ea47491d2f8291"}
Jul 23 00:34:50.234130 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:50.234099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn" event={"ID":"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b","Type":"ContainerStarted","Data":"c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0"}
Jul 23 00:34:50.234463 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:50.234193    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:34:50.252952 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:50.252909    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn" podStartSLOduration=0.639067824 podStartE2EDuration="2.252895786s" totalImagesPullingTime="1.613827962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:34:50.251143688 +0000 UTC m=+684.047022939" watchObservedRunningTime="2026-07-23 00:34:50.252895786 +0000 UTC m=+684.048775037"
Jul 23 00:34:52.209055 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.209006    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"]
Jul 23 00:34:52.212169 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.212152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:34:52.214386 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.214359    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bk5x8\""
Jul 23 00:34:52.227988 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.227963    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"]
Jul 23 00:34:52.269588 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.269557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrsc\" (UniqueName: \"kubernetes.io/projected/22c81994-2110-4398-9d00-bd1a0b714d3b-kube-api-access-qqrsc\") pod \"authorino-operator-75cc8d5cb7-6lksz\" (UID: \"22c81994-2110-4398-9d00-bd1a0b714d3b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:34:52.370981 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.370945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrsc\" (UniqueName: \"kubernetes.io/projected/22c81994-2110-4398-9d00-bd1a0b714d3b-kube-api-access-qqrsc\") pod \"authorino-operator-75cc8d5cb7-6lksz\" (UID: \"22c81994-2110-4398-9d00-bd1a0b714d3b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:34:52.379497 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.379466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrsc\" (UniqueName: \"kubernetes.io/projected/22c81994-2110-4398-9d00-bd1a0b714d3b-kube-api-access-qqrsc\") pod \"authorino-operator-75cc8d5cb7-6lksz\" (UID: \"22c81994-2110-4398-9d00-bd1a0b714d3b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:34:52.522123 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.522034    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:34:52.659217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:52.659196    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"]
Jul 23 00:34:52.661704 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:34:52.661677    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c81994_2110_4398_9d00_bd1a0b714d3b.slice/crio-e2e3a3f7810b08d6fffe2948c449b8602f04de2d3c195e4b3d89def647b24747 WatchSource:0}: Error finding container e2e3a3f7810b08d6fffe2948c449b8602f04de2d3c195e4b3d89def647b24747: Status 404 returned error can't find the container with id e2e3a3f7810b08d6fffe2948c449b8602f04de2d3c195e4b3d89def647b24747
Jul 23 00:34:53.253753 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:53.253717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz" event={"ID":"22c81994-2110-4398-9d00-bd1a0b714d3b","Type":"ContainerStarted","Data":"e2e3a3f7810b08d6fffe2948c449b8602f04de2d3c195e4b3d89def647b24747"}
Jul 23 00:34:55.265264 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:55.265182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz" event={"ID":"22c81994-2110-4398-9d00-bd1a0b714d3b","Type":"ContainerStarted","Data":"ab42183766462b527ab34265a665baa1b1df50d8e9052ffcd3cc2e325cc6256e"}
Jul 23 00:34:55.265609 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:55.265341    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:34:55.284899 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:34:55.284852    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz" podStartSLOduration=1.0243312470000001 podStartE2EDuration="3.284838033s" totalImagesPullingTime="2.260506786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:34:55.283807926 +0000 UTC m=+689.079687177" watchObservedRunningTime="2026-07-23 00:34:55.284838033 +0000 UTC m=+689.080717284"
Jul 23 00:35:01.241085 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:01.241043    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:06.272378 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:06.272349    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6lksz"
Jul 23 00:35:10.171934 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.171897    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"]
Jul 23 00:35:10.175677 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.175652    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.178120 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.178096    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 00:35:10.178251 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.178171    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 00:35:10.178251 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.178178    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nzwvk\""
Jul 23 00:35:10.183788 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.183760    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"]
Jul 23 00:35:10.324815 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.324779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f41e866-7519-434d-b271-ed797e19a27c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.325045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.324828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f41e866-7519-434d-b271-ed797e19a27c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.325045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.324898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpnc\" (UniqueName: \"kubernetes.io/projected/9f41e866-7519-434d-b271-ed797e19a27c-kube-api-access-8gpnc\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.426005 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.425924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f41e866-7519-434d-b271-ed797e19a27c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.426005 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.425972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f41e866-7519-434d-b271-ed797e19a27c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.426244 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.426021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpnc\" (UniqueName: \"kubernetes.io/projected/9f41e866-7519-434d-b271-ed797e19a27c-kube-api-access-8gpnc\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.426680 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.426654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f41e866-7519-434d-b271-ed797e19a27c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.428287 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.428267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f41e866-7519-434d-b271-ed797e19a27c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.433885 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.433863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpnc\" (UniqueName: \"kubernetes.io/projected/9f41e866-7519-434d-b271-ed797e19a27c-kube-api-access-8gpnc\") pod \"kuadrant-console-plugin-66b5774985-6snll\" (UID: \"9f41e866-7519-434d-b271-ed797e19a27c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.498251 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.498194    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"
Jul 23 00:35:10.635686 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:10.635658    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6snll"]
Jul 23 00:35:10.638175 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:35:10.638144    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f41e866_7519_434d_b271_ed797e19a27c.slice/crio-90eab914809c99270612de7f37b3a2c1bc4fea1f390d408f47ca9aad6febb27a WatchSource:0}: Error finding container 90eab914809c99270612de7f37b3a2c1bc4fea1f390d408f47ca9aad6febb27a: Status 404 returned error can't find the container with id 90eab914809c99270612de7f37b3a2c1bc4fea1f390d408f47ca9aad6febb27a
Jul 23 00:35:11.334177 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:11.334137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll" event={"ID":"9f41e866-7519-434d-b271-ed797e19a27c","Type":"ContainerStarted","Data":"90eab914809c99270612de7f37b3a2c1bc4fea1f390d408f47ca9aad6febb27a"}
Jul 23 00:35:20.226366 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.226288    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"]
Jul 23 00:35:20.233519 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.233492    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.236045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.236021    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hdqfk\""
Jul 23 00:35:20.241831 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.241770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"]
Jul 23 00:35:20.315050 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.315011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfw87\" (UniqueName: \"kubernetes.io/projected/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-kube-api-access-hfw87\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.315241 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.315142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.415973 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.415930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfw87\" (UniqueName: \"kubernetes.io/projected/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-kube-api-access-hfw87\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.416172 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.416020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.416398 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.416376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.426780 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.426751    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfw87\" (UniqueName: \"kubernetes.io/projected/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-kube-api-access-hfw87\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:20.547181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:20.547100    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:21.061895 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.061857    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"]
Jul 23 00:35:21.066377 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.066350    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"]
Jul 23 00:35:21.087061 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.087019    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"]
Jul 23 00:35:21.092280 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.092257    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.105463 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.105385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"]
Jul 23 00:35:21.108017 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.107982    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"]
Jul 23 00:35:21.108474 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.108397    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" containerName="manager" containerID="cri-o://c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0" gracePeriod=2
Jul 23 00:35:21.116442 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.116421    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"]
Jul 23 00:35:21.120213 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:21.120177    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:21.134847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.134814    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"]
Jul 23 00:35:21.135630 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.135607    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" containerName="manager"
Jul 23 00:35:21.135722 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.135634    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" containerName="manager"
Jul 23 00:35:21.135866 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.135847    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" containerName="manager"
Jul 23 00:35:21.146782 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.146758    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:21.149522 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:21.149490    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:21.150119 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.150095    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"]
Jul 23 00:35:21.224031 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.223992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694z6\" (UniqueName: \"kubernetes.io/projected/6830ae00-11ef-4b8b-97b8-cccd4787dce3-kube-api-access-694z6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8mphj\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.224207 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.224057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srcrp\" (UniqueName: \"kubernetes.io/projected/e4125696-b354-4636-a0e6-dcf25658df56-kube-api-access-srcrp\") pod \"limitador-operator-controller-manager-859d7b78bd-fc7ts\" (UID: \"e4125696-b354-4636-a0e6-dcf25658df56\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:21.224207 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.224168    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6830ae00-11ef-4b8b-97b8-cccd4787dce3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8mphj\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.325441 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.325360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6830ae00-11ef-4b8b-97b8-cccd4787dce3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8mphj\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.325832 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.325443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694z6\" (UniqueName: \"kubernetes.io/projected/6830ae00-11ef-4b8b-97b8-cccd4787dce3-kube-api-access-694z6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8mphj\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.325832 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.325466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srcrp\" (UniqueName: \"kubernetes.io/projected/e4125696-b354-4636-a0e6-dcf25658df56-kube-api-access-srcrp\") pod \"limitador-operator-controller-manager-859d7b78bd-fc7ts\" (UID: \"e4125696-b354-4636-a0e6-dcf25658df56\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:21.325832 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.325772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6830ae00-11ef-4b8b-97b8-cccd4787dce3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8mphj\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.333774 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.333744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-694z6\" (UniqueName: \"kubernetes.io/projected/6830ae00-11ef-4b8b-97b8-cccd4787dce3-kube-api-access-694z6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8mphj\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.333923 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.333805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srcrp\" (UniqueName: \"kubernetes.io/projected/e4125696-b354-4636-a0e6-dcf25658df56-kube-api-access-srcrp\") pod \"limitador-operator-controller-manager-859d7b78bd-fc7ts\" (UID: \"e4125696-b354-4636-a0e6-dcf25658df56\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:21.409895 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.409852    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:21.477167 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:21.477128    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:22.240280 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:22.240232    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 00:35:33.805502 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:33.805477    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:33.807936 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:33.807903    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:33.814023 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:33.813996    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"]
Jul 23 00:35:33.815630 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:35:33.815606    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4125696_b354_4636_a0e6_dcf25658df56.slice/crio-2c36d94005cdf323f15968da66d7e4f80e996d77d7c36be15e58af9fd7e68339 WatchSource:0}: Error finding container 2c36d94005cdf323f15968da66d7e4f80e996d77d7c36be15e58af9fd7e68339: Status 404 returned error can't find the container with id 2c36d94005cdf323f15968da66d7e4f80e996d77d7c36be15e58af9fd7e68339
Jul 23 00:35:33.845997 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:33.845976    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm6bk\" (UniqueName: \"kubernetes.io/projected/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b-kube-api-access-xm6bk\") pod \"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b\" (UID: \"8848fa06-6cc2-4e7e-bd1e-739e1d8e402b\") "
Jul 23 00:35:33.847940 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:33.847913    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b-kube-api-access-xm6bk" (OuterVolumeSpecName: "kube-api-access-xm6bk") pod "8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" (UID: "8848fa06-6cc2-4e7e-bd1e-739e1d8e402b"). InnerVolumeSpecName "kube-api-access-xm6bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:35:33.851724 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:35:33.851685    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6830ae00_11ef_4b8b_97b8_cccd4787dce3.slice/crio-f5ca0cff01cbe82d48ced235ff1f9a9442cbd3c80c5583abd12dd31c45b4c5e5 WatchSource:0}: Error finding container f5ca0cff01cbe82d48ced235ff1f9a9442cbd3c80c5583abd12dd31c45b4c5e5: Status 404 returned error can't find the container with id f5ca0cff01cbe82d48ced235ff1f9a9442cbd3c80c5583abd12dd31c45b4c5e5
Jul 23 00:35:33.853096 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:33.853047    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"]
Jul 23 00:35:33.946847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:33.946786    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6bk\" (UniqueName: \"kubernetes.io/projected/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b-kube-api-access-xm6bk\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:35:34.446593 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.446556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll" event={"ID":"9f41e866-7519-434d-b271-ed797e19a27c","Type":"ContainerStarted","Data":"1c16bc66fd6fa396b95ad8c41048acfefde4ae4717ef4fe7854e92a824843174"}
Jul 23 00:35:34.447887 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.447852    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj" event={"ID":"6830ae00-11ef-4b8b-97b8-cccd4787dce3","Type":"ContainerStarted","Data":"f5ca0cff01cbe82d48ced235ff1f9a9442cbd3c80c5583abd12dd31c45b4c5e5"}
Jul 23 00:35:34.448731 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:34.448704    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:34.449045 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.449027    2569 generic.go:356] "Generic (PLEG): container finished" podID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" containerID="c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0" exitCode=0
Jul 23 00:35:34.449114 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.449062    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:34.449114 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.449108    2569 scope.go:117] "RemoveContainer" containerID="c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0"
Jul 23 00:35:34.450615 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.450590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts" event={"ID":"e4125696-b354-4636-a0e6-dcf25658df56","Type":"ContainerStarted","Data":"8de7f26478d2c5e9eef5f9c2f1dd2cf482195afe4cd893f296a76494eb882ef7"}
Jul 23 00:35:34.450746 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.450642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts" event={"ID":"e4125696-b354-4636-a0e6-dcf25658df56","Type":"ContainerStarted","Data":"2c36d94005cdf323f15968da66d7e4f80e996d77d7c36be15e58af9fd7e68339"}
Jul 23 00:35:34.450746 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.450683    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:34.457222 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.457201    2569 scope.go:117] "RemoveContainer" containerID="c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0"
Jul 23 00:35:34.457536 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:34.457518    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0\": container with ID starting with c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0 not found: ID does not exist" containerID="c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0"
Jul 23 00:35:34.457589 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.457547    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0"} err="failed to get container status \"c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0\": rpc error: code = NotFound desc = could not find container \"c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0\": container with ID starting with c7fcbe40d81289f22f6df06dd327a4df8f4d45166defec3f2e79673fc98527b0 not found: ID does not exist"
Jul 23 00:35:34.463996 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.463959    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6snll" podStartSLOduration=1.36576458 podStartE2EDuration="24.463948863s" totalImagesPullingTime="23.098184283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:35:34.463721146 +0000 UTC m=+728.259600417" watchObservedRunningTime="2026-07-23 00:35:34.463948863 +0000 UTC m=+728.259828114"
Jul 23 00:35:34.465555 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:34.465534    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6kbn\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6kbn"
Jul 23 00:35:34.483181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.483136    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts" podStartSLOduration=13.483122945 podStartE2EDuration="13.483122945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:35:34.48236466 +0000 UTC m=+728.278243925" watchObservedRunningTime="2026-07-23 00:35:34.483122945 +0000 UTC m=+728.279002187"
Jul 23 00:35:34.746192 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:34.746120    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8848fa06-6cc2-4e7e-bd1e-739e1d8e402b" path="/var/lib/kubelet/pods/8848fa06-6cc2-4e7e-bd1e-739e1d8e402b/volumes"
Jul 23 00:35:36.188969 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:36.188918    2569 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-cpsrv_kuadrant-system_781b2c2a-61fe-4cef-ad03-5f1f32cc9744_0(04fc82efb456c0acf86ad84b2a9d0e00ec18804421b04ddeace22106184953c7): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-cpsrv to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 23 00:35:36.189365 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:36.189005    2569 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-cpsrv_kuadrant-system_781b2c2a-61fe-4cef-ad03-5f1f32cc9744_0(04fc82efb456c0acf86ad84b2a9d0e00ec18804421b04ddeace22106184953c7): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-cpsrv to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:36.461564 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.461481    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:36.463787 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:36.463746    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:36.471044 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.471015    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:36.473338 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:36.473305    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:36.570733 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.570700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfw87\" (UniqueName: \"kubernetes.io/projected/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-kube-api-access-hfw87\") pod \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") "
Jul 23 00:35:36.570901 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.570805    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-extensions-socket-volume\") pod \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\" (UID: \"781b2c2a-61fe-4cef-ad03-5f1f32cc9744\") "
Jul 23 00:35:36.571150 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.571121    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "781b2c2a-61fe-4cef-ad03-5f1f32cc9744" (UID: "781b2c2a-61fe-4cef-ad03-5f1f32cc9744"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:35:36.573131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.573101    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-kube-api-access-hfw87" (OuterVolumeSpecName: "kube-api-access-hfw87") pod "781b2c2a-61fe-4cef-ad03-5f1f32cc9744" (UID: "781b2c2a-61fe-4cef-ad03-5f1f32cc9744"). InnerVolumeSpecName "kube-api-access-hfw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:35:36.671632 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.671592    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-extensions-socket-volume\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:35:36.671632 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.671626    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfw87\" (UniqueName: \"kubernetes.io/projected/781b2c2a-61fe-4cef-ad03-5f1f32cc9744-kube-api-access-hfw87\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:35:36.745654 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:36.745564    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:36.748682 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:36.748653    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" path="/var/lib/kubelet/pods/781b2c2a-61fe-4cef-ad03-5f1f32cc9744/volumes"
Jul 23 00:35:37.466998 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:37.466965    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:37.469678 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:37.469649    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:37.471671 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:37.471641    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:38.473205 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:38.473170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj" event={"ID":"6830ae00-11ef-4b8b-97b8-cccd4787dce3","Type":"ContainerStarted","Data":"f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb"}
Jul 23 00:35:38.473616 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:38.473238    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:35:38.475298 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:35:38.475267    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cpsrv\" is forbidden: User \"system:node:ip-10-0-139-138.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-138.ec2.internal' and this object" podUID="781b2c2a-61fe-4cef-ad03-5f1f32cc9744" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cpsrv"
Jul 23 00:35:38.503650 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:38.503565    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj" podStartSLOduration=13.107808131 podStartE2EDuration="17.503549164s" totalImagesPullingTime="4.395741033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:35:38.50132756 +0000 UTC m=+732.297206855" watchObservedRunningTime="2026-07-23 00:35:38.503549164 +0000 UTC m=+732.299428415"
Jul 23 00:35:45.459213 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:45.459179    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fc7ts"
Jul 23 00:35:49.479706 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:35:49.479673    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:36:01.521620 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.521582    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"]
Jul 23 00:36:01.522110 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.521865    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj" podUID="6830ae00-11ef-4b8b-97b8-cccd4787dce3" containerName="manager" containerID="cri-o://f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb" gracePeriod=10
Jul 23 00:36:01.779807 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.779750    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:36:01.897965 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.897928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6830ae00-11ef-4b8b-97b8-cccd4787dce3-extensions-socket-volume\") pod \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") "
Jul 23 00:36:01.898114 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.897979    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-694z6\" (UniqueName: \"kubernetes.io/projected/6830ae00-11ef-4b8b-97b8-cccd4787dce3-kube-api-access-694z6\") pod \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\" (UID: \"6830ae00-11ef-4b8b-97b8-cccd4787dce3\") "
Jul 23 00:36:01.898458 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.898433    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6830ae00-11ef-4b8b-97b8-cccd4787dce3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6830ae00-11ef-4b8b-97b8-cccd4787dce3" (UID: "6830ae00-11ef-4b8b-97b8-cccd4787dce3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:36:01.900138 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.900108    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6830ae00-11ef-4b8b-97b8-cccd4787dce3-kube-api-access-694z6" (OuterVolumeSpecName: "kube-api-access-694z6") pod "6830ae00-11ef-4b8b-97b8-cccd4787dce3" (UID: "6830ae00-11ef-4b8b-97b8-cccd4787dce3"). InnerVolumeSpecName "kube-api-access-694z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:36:01.998857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.998830    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6830ae00-11ef-4b8b-97b8-cccd4787dce3-extensions-socket-volume\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:36:01.998857 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:01.998854    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694z6\" (UniqueName: \"kubernetes.io/projected/6830ae00-11ef-4b8b-97b8-cccd4787dce3-kube-api-access-694z6\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:36:02.580946 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.580908    2569 generic.go:356] "Generic (PLEG): container finished" podID="6830ae00-11ef-4b8b-97b8-cccd4787dce3" containerID="f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb" exitCode=0
Jul 23 00:36:02.581343 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.580950    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"
Jul 23 00:36:02.581343 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.580991    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj" event={"ID":"6830ae00-11ef-4b8b-97b8-cccd4787dce3","Type":"ContainerDied","Data":"f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb"}
Jul 23 00:36:02.581343 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.581055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj" event={"ID":"6830ae00-11ef-4b8b-97b8-cccd4787dce3","Type":"ContainerDied","Data":"f5ca0cff01cbe82d48ced235ff1f9a9442cbd3c80c5583abd12dd31c45b4c5e5"}
Jul 23 00:36:02.581343 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.581100    2569 scope.go:117] "RemoveContainer" containerID="f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb"
Jul 23 00:36:02.590334 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.590317    2569 scope.go:117] "RemoveContainer" containerID="f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb"
Jul 23 00:36:02.590592 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:36:02.590575    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb\": container with ID starting with f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb not found: ID does not exist" containerID="f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb"
Jul 23 00:36:02.590638 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.590601    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb"} err="failed to get container status \"f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb\": rpc error: code = NotFound desc = could not find container \"f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb\": container with ID starting with f76626f0b9f37efee17233b98274a638f853d3b352a3bc061bd40c8df91555eb not found: ID does not exist"
Jul 23 00:36:02.608457 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.608435    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"]
Jul 23 00:36:02.613932 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.613913    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8mphj"]
Jul 23 00:36:02.745507 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:02.745480    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6830ae00-11ef-4b8b-97b8-cccd4787dce3" path="/var/lib/kubelet/pods/6830ae00-11ef-4b8b-97b8-cccd4787dce3/volumes"
Jul 23 00:36:10.609893 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.609857    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:36:10.610298 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.610195    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6830ae00-11ef-4b8b-97b8-cccd4787dce3" containerName="manager"
Jul 23 00:36:10.610298 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.610208    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6830ae00-11ef-4b8b-97b8-cccd4787dce3" containerName="manager"
Jul 23 00:36:10.610298 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.610273    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6830ae00-11ef-4b8b-97b8-cccd4787dce3" containerName="manager"
Jul 23 00:36:10.680193 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.680158    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:36:10.680366 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.680203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:36:10.680366 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.680279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.682545 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.682523    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 00:36:10.777749 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.777716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrc6f\" (UniqueName: \"kubernetes.io/projected/20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e-kube-api-access-nrc6f\") pod \"limitador-limitador-54d6d988d5-6tpst\" (UID: \"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.777917 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.777794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e-config-file\") pod \"limitador-limitador-54d6d988d5-6tpst\" (UID: \"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.879061 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.878981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e-config-file\") pod \"limitador-limitador-54d6d988d5-6tpst\" (UID: \"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.879061 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.879048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrc6f\" (UniqueName: \"kubernetes.io/projected/20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e-kube-api-access-nrc6f\") pod \"limitador-limitador-54d6d988d5-6tpst\" (UID: \"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.879640 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.879617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e-config-file\") pod \"limitador-limitador-54d6d988d5-6tpst\" (UID: \"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.886782 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.886762    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrc6f\" (UniqueName: \"kubernetes.io/projected/20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e-kube-api-access-nrc6f\") pod \"limitador-limitador-54d6d988d5-6tpst\" (UID: \"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:10.990755 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:10.990720    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:11.129575 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.129443    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:36:11.131935 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:36:11.131905    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d8b0f3_dcb4_4b7b_b85f_2a69b9c94f1e.slice/crio-a832617508a38edebe16ed21055729240f71dd7ba6043a071cc811332e7a4c16 WatchSource:0}: Error finding container a832617508a38edebe16ed21055729240f71dd7ba6043a071cc811332e7a4c16: Status 404 returned error can't find the container with id a832617508a38edebe16ed21055729240f71dd7ba6043a071cc811332e7a4c16
Jul 23 00:36:11.308421 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.308383    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-p98bx"]
Jul 23 00:36:11.312972 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.312954    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:11.315453 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.315421    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sx4qg\""
Jul 23 00:36:11.317368 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.317346    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p98bx"]
Jul 23 00:36:11.484727 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.484639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvm7\" (UniqueName: \"kubernetes.io/projected/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b-kube-api-access-qbvm7\") pod \"authorino-75d556bd77-p98bx\" (UID: \"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b\") " pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:11.585212 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.585149    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvm7\" (UniqueName: \"kubernetes.io/projected/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b-kube-api-access-qbvm7\") pod \"authorino-75d556bd77-p98bx\" (UID: \"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b\") " pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:11.594254 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.594194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvm7\" (UniqueName: \"kubernetes.io/projected/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b-kube-api-access-qbvm7\") pod \"authorino-75d556bd77-p98bx\" (UID: \"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b\") " pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:11.622720 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.622683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:11.633747 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.633519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst" event={"ID":"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e","Type":"ContainerStarted","Data":"a832617508a38edebe16ed21055729240f71dd7ba6043a071cc811332e7a4c16"}
Jul 23 00:36:11.813986 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:11.813918    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p98bx"]
Jul 23 00:36:11.815390 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:36:11.815348    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedaa8499_98ad_4edf_8cf2_b7a05bbaab4b.slice/crio-29fc86817c0580177df00b084912c202f2c1130a727bfbdeba7fb4980f55568c WatchSource:0}: Error finding container 29fc86817c0580177df00b084912c202f2c1130a727bfbdeba7fb4980f55568c: Status 404 returned error can't find the container with id 29fc86817c0580177df00b084912c202f2c1130a727bfbdeba7fb4980f55568c
Jul 23 00:36:12.643496 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:12.643453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p98bx" event={"ID":"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b","Type":"ContainerStarted","Data":"29fc86817c0580177df00b084912c202f2c1130a727bfbdeba7fb4980f55568c"}
Jul 23 00:36:15.663217 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:15.663172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst" event={"ID":"20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e","Type":"ContainerStarted","Data":"0fbf9a0ac8eecab5c1ee2d74ea2be5457633653fa2eb7e52b2c11b37e84f48aa"}
Jul 23 00:36:15.663644 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:15.663378    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:15.680390 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:15.680332    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst" podStartSLOduration=1.278443884 podStartE2EDuration="5.680316677s" totalImagesPullingTime="4.401872793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:36:15.678864073 +0000 UTC m=+769.474743326" watchObservedRunningTime="2026-07-23 00:36:15.680316677 +0000 UTC m=+769.476195989"
Jul 23 00:36:15.694165 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:15.694108    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-p98bx" podStartSLOduration=0.917683083 podStartE2EDuration="4.694061201s" totalImagesPullingTime="3.776378118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:36:15.692484969 +0000 UTC m=+769.488364221" watchObservedRunningTime="2026-07-23 00:36:15.694061201 +0000 UTC m=+769.489940452"
Jul 23 00:36:16.671003 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:16.670968    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p98bx" event={"ID":"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b","Type":"ContainerStarted","Data":"fe4eee91e58cd764184e923ab194bfe5aaea15e5eab4420e6af1b8f6bf312863"}
Jul 23 00:36:26.672690 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:26.672662    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6tpst"
Jul 23 00:36:42.659024 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:42.658357    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p98bx"]
Jul 23 00:36:42.659024 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:42.658635    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-p98bx" podUID="edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" containerName="authorino" containerID="cri-o://fe4eee91e58cd764184e923ab194bfe5aaea15e5eab4420e6af1b8f6bf312863" gracePeriod=30
Jul 23 00:36:42.794263 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:42.794228    2569 generic.go:356] "Generic (PLEG): container finished" podID="edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" containerID="fe4eee91e58cd764184e923ab194bfe5aaea15e5eab4420e6af1b8f6bf312863" exitCode=0
Jul 23 00:36:42.794415 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:42.794301    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p98bx" event={"ID":"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b","Type":"ContainerDied","Data":"fe4eee91e58cd764184e923ab194bfe5aaea15e5eab4420e6af1b8f6bf312863"}
Jul 23 00:36:42.909907 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:42.909848    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:43.070985 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.070946    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbvm7\" (UniqueName: \"kubernetes.io/projected/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b-kube-api-access-qbvm7\") pod \"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b\" (UID: \"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b\") "
Jul 23 00:36:43.072989 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.072954    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b-kube-api-access-qbvm7" (OuterVolumeSpecName: "kube-api-access-qbvm7") pod "edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" (UID: "edaa8499-98ad-4edf-8cf2-b7a05bbaab4b"). InnerVolumeSpecName "kube-api-access-qbvm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:36:43.172188 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.172104    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvm7\" (UniqueName: \"kubernetes.io/projected/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b-kube-api-access-qbvm7\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:36:43.800413 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.800389    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p98bx"
Jul 23 00:36:43.800833 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.800432    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p98bx" event={"ID":"edaa8499-98ad-4edf-8cf2-b7a05bbaab4b","Type":"ContainerDied","Data":"29fc86817c0580177df00b084912c202f2c1130a727bfbdeba7fb4980f55568c"}
Jul 23 00:36:43.800833 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.800477    2569 scope.go:117] "RemoveContainer" containerID="fe4eee91e58cd764184e923ab194bfe5aaea15e5eab4420e6af1b8f6bf312863"
Jul 23 00:36:43.820308 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.820283    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p98bx"]
Jul 23 00:36:43.824482 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:43.824457    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p98bx"]
Jul 23 00:36:44.744980 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:36:44.744946    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" path="/var/lib/kubelet/pods/edaa8499-98ad-4edf-8cf2-b7a05bbaab4b/volumes"
Jul 23 00:37:24.598161 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:37:24.598086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:37:56.178958 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:37:56.178927    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:38:00.981669 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:00.981627    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:38:12.170821 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.170788    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"]
Jul 23 00:38:12.171193 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.171182    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" containerName="authorino"
Jul 23 00:38:12.171235 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.171197    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" containerName="authorino"
Jul 23 00:38:12.171271 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.171267    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="edaa8499-98ad-4edf-8cf2-b7a05bbaab4b" containerName="authorino"
Jul 23 00:38:12.174510 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.174493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.178616 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.178592    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 00:38:12.178735 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.178625    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 23 00:38:12.179393 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.179373    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jtgsc\""
Jul 23 00:38:12.179492 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.179409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 00:38:12.185009 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.184991    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"]
Jul 23 00:38:12.227269 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.227237    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.227407 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.227282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.227407 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.227307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.227407 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.227349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zlf\" (UniqueName: \"kubernetes.io/projected/373220b2-7e47-4921-8d87-906ff774597b-kube-api-access-v4zlf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.227407 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.227404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.227575 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.227474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373220b2-7e47-4921-8d87-906ff774597b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.327917 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.327883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328040 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.327923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373220b2-7e47-4921-8d87-906ff774597b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328040 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.327996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328040 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.328028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328289 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.328087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328289 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.328124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zlf\" (UniqueName: \"kubernetes.io/projected/373220b2-7e47-4921-8d87-906ff774597b-kube-api-access-v4zlf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328441 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.328412    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328441 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.328437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.328623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.328482    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.330177 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.330156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373220b2-7e47-4921-8d87-906ff774597b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.330304 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.330288    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373220b2-7e47-4921-8d87-906ff774597b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.335877 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.335854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zlf\" (UniqueName: \"kubernetes.io/projected/373220b2-7e47-4921-8d87-906ff774597b-kube-api-access-v4zlf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6n22g\" (UID: \"373220b2-7e47-4921-8d87-906ff774597b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.514465 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.514384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:12.643393 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:12.643371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"]
Jul 23 00:38:12.645606 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:38:12.645579    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373220b2_7e47_4921_8d87_906ff774597b.slice/crio-876894015c809c292d537ac60647c7e67f477b4164483c8d9db8480d96f3649e WatchSource:0}: Error finding container 876894015c809c292d537ac60647c7e67f477b4164483c8d9db8480d96f3649e: Status 404 returned error can't find the container with id 876894015c809c292d537ac60647c7e67f477b4164483c8d9db8480d96f3649e
Jul 23 00:38:13.197237 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:13.197194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g" event={"ID":"373220b2-7e47-4921-8d87-906ff774597b","Type":"ContainerStarted","Data":"876894015c809c292d537ac60647c7e67f477b4164483c8d9db8480d96f3649e"}
Jul 23 00:38:13.385740 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:13.385699    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:38:14.203259 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:14.203219    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g" event={"ID":"373220b2-7e47-4921-8d87-906ff774597b","Type":"ContainerStarted","Data":"e104dc8074a155b58ee66533c0c1d4538664829d68e5dbcd96546c5312eb0352"}
Jul 23 00:38:14.222733 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:14.222686    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g" podStartSLOduration=1.255793193 podStartE2EDuration="2.222667177s" totalImagesPullingTime="966.873984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:38:14.22136514 +0000 UTC m=+888.017244392" watchObservedRunningTime="2026-07-23 00:38:14.222667177 +0000 UTC m=+888.018546484"
Jul 23 00:38:22.514541 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:22.514461    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:22.514541 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:22.514533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:22.527084 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:22.527046    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:23.255115 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:23.255086    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6n22g"
Jul 23 00:38:25.580169 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.580124    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"]
Jul 23 00:38:25.583540 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.583516    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.586022 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.586003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 23 00:38:25.592333 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.592313    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"]
Jul 23 00:38:25.642347 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.642322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.642347 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.642353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xmp\" (UniqueName: \"kubernetes.io/projected/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-kube-api-access-t2xmp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.642513 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.642377    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.642513 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.642423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.642581 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.642509    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.642581 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.642555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743501 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xmp\" (UniqueName: \"kubernetes.io/projected/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-kube-api-access-t2xmp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743623 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743922 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.743990 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.744047 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.743997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.745655 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.745637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.745929 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.745912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.751284 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.751255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xmp\" (UniqueName: \"kubernetes.io/projected/9ff7b2b9-c65b-443a-a179-c0baf8d617c6-kube-api-access-t2xmp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7\" (UID: \"9ff7b2b9-c65b-443a-a179-c0baf8d617c6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:25.894331 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:25.894304    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:26.024880 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.024852    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"]
Jul 23 00:38:26.027611 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:38:26.027583    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff7b2b9_c65b_443a_a179_c0baf8d617c6.slice/crio-bd1a3ebd80501aab1b38281206cd5aa33c9aeb82c166eca7ad318eedd62eaf46 WatchSource:0}: Error finding container bd1a3ebd80501aab1b38281206cd5aa33c9aeb82c166eca7ad318eedd62eaf46: Status 404 returned error can't find the container with id bd1a3ebd80501aab1b38281206cd5aa33c9aeb82c166eca7ad318eedd62eaf46
Jul 23 00:38:26.263677 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.263652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7" event={"ID":"9ff7b2b9-c65b-443a-a179-c0baf8d617c6","Type":"ContainerStarted","Data":"f1b9852161bada931e855901f21b05e2f7796545325904a2779135fe28bdd2b9"}
Jul 23 00:38:26.263778 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.263701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7" event={"ID":"9ff7b2b9-c65b-443a-a179-c0baf8d617c6","Type":"ContainerStarted","Data":"bd1a3ebd80501aab1b38281206cd5aa33c9aeb82c166eca7ad318eedd62eaf46"}
Jul 23 00:38:26.626876 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.626847    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:38:26.627358 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.627272    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:38:26.633204 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.633181    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:38:26.633456 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:26.633436    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:38:27.083847 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:27.083770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:38:27.285251 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:27.285200    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7" podStartSLOduration=2.104333201 podStartE2EDuration="2.285180894s" totalImagesPullingTime="180.847693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:38:27.283818891 +0000 UTC m=+901.079698142" watchObservedRunningTime="2026-07-23 00:38:27.285180894 +0000 UTC m=+901.081060145"
Jul 23 00:38:35.895265 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:35.895217    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:35.897708 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:35.895293    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:35.908248 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:35.908220    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:36.318193 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:36.318111    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7"
Jul 23 00:38:53.893409 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:53.893374    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"]
Jul 23 00:38:53.896821 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:53.896803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:53.899304 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:53.899285    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 23 00:38:53.904542 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:53.904521    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"]
Jul 23 00:38:54.080809 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.080777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.080809 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.080811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.081008 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.080837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4fc\" (UniqueName: \"kubernetes.io/projected/42b4bdd5-b5aa-4793-bbad-84812e79f96c-kube-api-access-bx4fc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.081008 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.080934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.081008 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.080986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b4bdd5-b5aa-4793-bbad-84812e79f96c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.081157 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.081031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182100 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182100 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b4bdd5-b5aa-4793-bbad-84812e79f96c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182100 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182412 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182412 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182164    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182412 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4fc\" (UniqueName: \"kubernetes.io/projected/42b4bdd5-b5aa-4793-bbad-84812e79f96c-kube-api-access-bx4fc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182583 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182460    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182583 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182494    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.182674 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.182588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.184446 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.184417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b4bdd5-b5aa-4793-bbad-84812e79f96c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.184678 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.184661    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b4bdd5-b5aa-4793-bbad-84812e79f96c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.190094 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.190053    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4fc\" (UniqueName: \"kubernetes.io/projected/42b4bdd5-b5aa-4793-bbad-84812e79f96c-kube-api-access-bx4fc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n9cdn\" (UID: \"42b4bdd5-b5aa-4793-bbad-84812e79f96c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.208938 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.208919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:38:54.336736 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.336710    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"]
Jul 23 00:38:54.338617 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:38:54.338584    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b4bdd5_b5aa_4793_bbad_84812e79f96c.slice/crio-8b6223f80c5bceec4b95320c44f70674bad880e19119a79cb62c32707adbaf21 WatchSource:0}: Error finding container 8b6223f80c5bceec4b95320c44f70674bad880e19119a79cb62c32707adbaf21: Status 404 returned error can't find the container with id 8b6223f80c5bceec4b95320c44f70674bad880e19119a79cb62c32707adbaf21
Jul 23 00:38:54.391723 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:54.391685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn" event={"ID":"42b4bdd5-b5aa-4793-bbad-84812e79f96c","Type":"ContainerStarted","Data":"8b6223f80c5bceec4b95320c44f70674bad880e19119a79cb62c32707adbaf21"}
Jul 23 00:38:55.397183 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:55.397151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn" event={"ID":"42b4bdd5-b5aa-4793-bbad-84812e79f96c","Type":"ContainerStarted","Data":"f719bd3f6af2c15abc70aadce21abd9c74c968431dd3a9e7bc206e15e7b0208d"}
Jul 23 00:38:55.416869 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:55.416816    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn" podStartSLOduration=2.207593227 podStartE2EDuration="2.416801418s" totalImagesPullingTime="209.208191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:38:55.414345714 +0000 UTC m=+929.210224965" watchObservedRunningTime="2026-07-23 00:38:55.416801418 +0000 UTC m=+929.212680670"
Jul 23 00:38:56.689300 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:38:56.689266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:39:04.209713 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:04.209658    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:39:04.210228 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:04.209751    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:39:04.224658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:04.224629    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:39:04.448444 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:04.448415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n9cdn"
Jul 23 00:39:05.474080 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.474037    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"]
Jul 23 00:39:05.477652 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.477629    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.480514 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.480493    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 00:39:05.491252 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.491226    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"]
Jul 23 00:39:05.566642 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.566611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shn94\" (UniqueName: \"kubernetes.io/projected/7ba09c19-0eb1-4b30-a842-6363172d2924-kube-api-access-shn94\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.566772 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.566646    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.566772 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.566689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.566848 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.566796    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.566848 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.566818    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba09c19-0eb1-4b30-a842-6363172d2924-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.566912 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.566846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667459 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667615 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667615 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667730 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba09c19-0eb1-4b30-a842-6363172d2924-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667730 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667842 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shn94\" (UniqueName: \"kubernetes.io/projected/7ba09c19-0eb1-4b30-a842-6363172d2924-kube-api-access-shn94\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.667933 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.667912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.668041 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.668014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.668146 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.668041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.669742 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.669718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba09c19-0eb1-4b30-a842-6363172d2924-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.670002 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.669987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba09c19-0eb1-4b30-a842-6363172d2924-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.675381 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.675360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shn94\" (UniqueName: \"kubernetes.io/projected/7ba09c19-0eb1-4b30-a842-6363172d2924-kube-api-access-shn94\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7\" (UID: \"7ba09c19-0eb1-4b30-a842-6363172d2924\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.786597 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.786512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:05.920341 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:05.920313    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"]
Jul 23 00:39:05.922400 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:39:05.922371    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ba09c19_0eb1_4b30_a842_6363172d2924.slice/crio-68873a8575bfabab9b66af72459eecd6c099851030370a645a509d2bdc090b12 WatchSource:0}: Error finding container 68873a8575bfabab9b66af72459eecd6c099851030370a645a509d2bdc090b12: Status 404 returned error can't find the container with id 68873a8575bfabab9b66af72459eecd6c099851030370a645a509d2bdc090b12
Jul 23 00:39:06.448559 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:06.448469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7" event={"ID":"7ba09c19-0eb1-4b30-a842-6363172d2924","Type":"ContainerStarted","Data":"fd82c2ada5e5018a528921549e7d70b04fec4ed2b922ccedb67955427f6a2642"}
Jul 23 00:39:06.448559 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:06.448522    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7" event={"ID":"7ba09c19-0eb1-4b30-a842-6363172d2924","Type":"ContainerStarted","Data":"68873a8575bfabab9b66af72459eecd6c099851030370a645a509d2bdc090b12"}
Jul 23 00:39:06.468876 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:06.468833    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7" podStartSLOduration=1.251504118 podStartE2EDuration="1.468818444s" totalImagesPullingTime="217.314326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:39:06.468029184 +0000 UTC m=+940.263908434" watchObservedRunningTime="2026-07-23 00:39:06.468818444 +0000 UTC m=+940.264697693"
Jul 23 00:39:07.885494 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:07.885459    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:39:15.786745 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:15.786702    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:15.787160 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:15.786764    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:15.799656 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:15.799627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:39:16.507367 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:39:16.507337    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7"
Jul 23 00:40:24.690881 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:40:24.690799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:40:35.085657 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:40:35.085623    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:40:45.279142 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:40:45.279108    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:42:29.230650 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:42:29.230613    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:42:40.680205 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:42:40.680172    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:43:14.285758 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:14.285680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:43:24.277721 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:24.277687    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:43:26.656590 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:26.656560    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:43:26.659127 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:26.659102    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:43:26.662896 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:26.662872    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:43:26.665460 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:26.665443    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:43:32.983186 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:32.983148    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:43:42.281769 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:42.281719    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:43:53.001156 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:43:53.001122    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:44:02.288609 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:44:02.288574    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:44:13.292229 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:44:13.292181    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:44:23.277605 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:44:23.277524    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:44:45.681284 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:44:45.681251    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:44:57.485142 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:44:57.485100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:45:19.980658 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:45:19.980618    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:45:32.082846 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:45:32.082772    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:45:48.579571 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:45:48.579528    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:45:57.686311 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:45:57.686275    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:46:07.189550 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:46:07.189515    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:46:16.584131 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:46:16.584095    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:46:25.880987 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:46:25.880952    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:48:14.293163 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:14.293120    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:48:23.579115 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:23.579062    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:48:26.684911 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:26.684883    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:48:26.691676 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:26.691647    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:48:26.694062 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:26.694039    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:48:26.700037 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:26.700013    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:48:36.088002 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:36.087965    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:48:48.301364 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:48.301326    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:48:57.875218 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:48:57.875175    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:49:08.183507 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:49:08.183471    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:49:18.686148 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:49:18.686111    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:49:28.789092 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:49:28.789032    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:49:39.285683 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:49:39.285648    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:49:48.878755 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:49:48.878721    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:49:59.981387 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:49:59.981348    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:50:23.583976 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:23.583895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:50:31.290560 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:31.290525    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:50:40.471421 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.471382    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"]
Jul 23 00:50:40.476120 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.476095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.478625 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.478604    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c49d048f-kserve-self-signed-certs\""
Jul 23 00:50:40.482565 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.482540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"]
Jul 23 00:50:40.588855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.588820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-dshm\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.588855 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.588859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-model-cache\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.589099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.588892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tmp-dir\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.589099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.588979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-home\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.589099 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.589011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tls-certs\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.589233 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.589098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lscs\" (UniqueName: \"kubernetes.io/projected/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-kube-api-access-4lscs\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.689704 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.689671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-dshm\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.689704 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.689707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-model-cache\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.689925 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.689732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tmp-dir\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.689925 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.689758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-home\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.689925 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.689777    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tls-certs\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.689925 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.689826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lscs\" (UniqueName: \"kubernetes.io/projected/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-kube-api-access-4lscs\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.690299 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.690258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tmp-dir\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.690299 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.690273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-model-cache\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.690470 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.690305    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-home\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.692021 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.691994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-dshm\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.692296 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.692278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tls-certs\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.697722 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.697701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lscs\" (UniqueName: \"kubernetes.io/projected/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-kube-api-access-4lscs\") pod \"e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") " pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.786604 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.786524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:40.916851 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:40.916800    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"]
Jul 23 00:50:40.919141 ip-10-0-139-138 kubenswrapper[2569]: W0723 00:50:40.919115    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b7ed8c6_70a3_4af9_abb4_321c9722f8b3.slice/crio-cab09bb606a6b62660df610efcde53912ffd5903990bd6709096a3cce8dd250a WatchSource:0}: Error finding container cab09bb606a6b62660df610efcde53912ffd5903990bd6709096a3cce8dd250a: Status 404 returned error can't find the container with id cab09bb606a6b62660df610efcde53912ffd5903990bd6709096a3cce8dd250a
Jul 23 00:50:41.618644 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:41.618610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7" event={"ID":"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3","Type":"ContainerStarted","Data":"a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99"}
Jul 23 00:50:41.619021 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:41.618652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7" event={"ID":"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3","Type":"ContainerStarted","Data":"cab09bb606a6b62660df610efcde53912ffd5903990bd6709096a3cce8dd250a"}
Jul 23 00:50:41.635734 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:41.635689    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7" podStartSLOduration=1.6356727 podStartE2EDuration="1.6356727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:50:41.634322023 +0000 UTC m=+1635.430201275" watchObservedRunningTime="2026-07-23 00:50:41.6356727 +0000 UTC m=+1635.431551997"
Jul 23 00:50:43.676750 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:43.676713    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:50:50.786753 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:50.786720    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:50.787170 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:50.786993    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:50.799539 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:50.799517    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:50:51.675039 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:50:51.675013    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:51:19.075231 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:19.075196    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"]
Jul 23 00:51:22.081246 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:22.081214    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:51:34.095581 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.095540    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7" podUID="4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" containerName="main" containerID="cri-o://a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99" gracePeriod=45
Jul 23 00:51:34.347568 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.347504    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:51:34.480481 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480454    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tls-certs\") pod \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") "
Jul 23 00:51:34.480642 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480491    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lscs\" (UniqueName: \"kubernetes.io/projected/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-kube-api-access-4lscs\") pod \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") "
Jul 23 00:51:34.480642 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480529    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tmp-dir\") pod \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") "
Jul 23 00:51:34.480754 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480638    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-model-cache\") pod \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") "
Jul 23 00:51:34.480754 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480728    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-dshm\") pod \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") "
Jul 23 00:51:34.480868 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" (UID: "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:51:34.480868 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-home\") pod \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\" (UID: \"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3\") "
Jul 23 00:51:34.480981 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480881    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" (UID: "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:51:34.480981 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.480919    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-home" (OuterVolumeSpecName: "home") pod "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" (UID: "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:51:34.481181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.481160    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:51:34.481181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.481178    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:51:34.481181 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.481186    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:51:34.482741 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.482708    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-dshm" (OuterVolumeSpecName: "dshm") pod "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" (UID: "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:51:34.482835 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.482768    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" (UID: "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:51:34.482835 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.482771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-kube-api-access-4lscs" (OuterVolumeSpecName: "kube-api-access-4lscs") pod "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" (UID: "4b7ed8c6-70a3-4af9-abb4-321c9722f8b3"). InnerVolumeSpecName "kube-api-access-4lscs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:51:34.581688 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.581663    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:51:34.581688 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.581687    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lscs\" (UniqueName: \"kubernetes.io/projected/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-kube-api-access-4lscs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:51:34.581851 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.581697    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 00:51:34.867059 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.867024    2569 generic.go:356] "Generic (PLEG): container finished" podID="4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" containerID="a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99" exitCode=0
Jul 23 00:51:34.867258 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.867064    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"
Jul 23 00:51:34.867258 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.867113    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7" event={"ID":"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3","Type":"ContainerDied","Data":"a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99"}
Jul 23 00:51:34.867258 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.867160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7" event={"ID":"4b7ed8c6-70a3-4af9-abb4-321c9722f8b3","Type":"ContainerDied","Data":"cab09bb606a6b62660df610efcde53912ffd5903990bd6709096a3cce8dd250a"}
Jul 23 00:51:34.867258 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.867182    2569 scope.go:117] "RemoveContainer" containerID="a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99"
Jul 23 00:51:34.875803 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.875782    2569 scope.go:117] "RemoveContainer" containerID="a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99"
Jul 23 00:51:34.876096 ip-10-0-139-138 kubenswrapper[2569]: E0723 00:51:34.876061    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99\": container with ID starting with a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99 not found: ID does not exist" containerID="a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99"
Jul 23 00:51:34.876162 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.876104    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99"} err="failed to get container status \"a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99\": rpc error: code = NotFound desc = could not find container \"a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99\": container with ID starting with a9343fdaf460729fd6c689568a50dabd09c5a19262750e3f7eb186d3345b0d99 not found: ID does not exist"
Jul 23 00:51:34.884346 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.884327    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"]
Jul 23 00:51:34.885944 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:34.885925    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c49d048f-kserve-78dcbd859-428b7"]
Jul 23 00:51:36.387120 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:36.387086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:51:36.746516 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:36.746443    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" path="/var/lib/kubelet/pods/4b7ed8c6-70a3-4af9-abb4-321c9722f8b3/volumes"
Jul 23 00:51:42.977597 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:42.977563    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:51:58.984526 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:51:58.984437    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:52:09.383871 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:52:09.383833    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:52:18.579488 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:52:18.579450    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:52:27.699466 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:52:27.699426    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:52:38.126494 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:52:38.126452    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:52:47.582653 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:52:47.582615    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:53:00.488171 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:00.488133    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:53:09.691238 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:09.691200    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:53:19.475973 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:19.475890    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:53:26.712372 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:26.712342    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:53:26.718994 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:26.718968    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:53:26.721748 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:26.721724    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:53:26.727914 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:26.727892    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:53:31.686438 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:31.686404    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:53:42.585140 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:42.585097    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:53:52.185854 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:53:52.185811    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:54:04.324618 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:54:04.324579    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:54:16.183253 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:54:16.183213    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:54:28.479582 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:54:28.479540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:54:42.182698 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:54:42.182663    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:54:54.586908 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:54:54.586823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:55:12.378670 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:55:12.378633    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:55:45.680637 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:55:45.680600    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:55:58.882513 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:55:58.882480    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:56:29.874278 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:56:29.874191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:56:42.476270 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:56:42.476229    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:57:13.686882 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:57:13.686844    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:57:26.079237 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:57:26.079195    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:57:57.873693 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:57:57.873613    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:58:26.747438 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:58:26.747408    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:58:26.753705 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:58:26.753678    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:58:26.754227 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:58:26.754206    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 00:58:26.761592 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:58:26.761573    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 00:59:24.681032 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:59:24.680935    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:59:35.183137 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:59:35.183096    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 00:59:46.286742 ip-10-0-139-138 kubenswrapper[2569]: I0723 00:59:46.286701    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:00:04.578249 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:04.578206    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:00:41.378373 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:41.378337    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:00:52.308616 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.308534    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"]
Jul 23 01:00:52.308993 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.308896    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" containerName="main"
Jul 23 01:00:52.308993 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.308907    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" containerName="main"
Jul 23 01:00:52.308993 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.308980    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7ed8c6-70a3-4af9-abb4-321c9722f8b3" containerName="main"
Jul 23 01:00:52.312123 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.312095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.315198 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.315172    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0828012d\"/\"openshift-service-ca.crt\""
Jul 23 01:00:52.315992 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.315961    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0828012d\"/\"e2e-lifecycle-model-0828012d-kserve-self-signed-certs\""
Jul 23 01:00:52.315992 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.315965    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0828012d\"/\"kube-root-ca.crt\""
Jul 23 01:00:52.316204 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.316103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0828012d\"/\"default-dockercfg-zflbq\""
Jul 23 01:00:52.321400 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.321379    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"]
Jul 23 01:00:52.406887 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.406847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-model-cache\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.406887 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.406884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-dshm\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.407163 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.406917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-home\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.407163 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.406941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvm9v\" (UniqueName: \"kubernetes.io/projected/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-kube-api-access-tvm9v\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.407163 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.407008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tls-certs\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.407163 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.407090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tmp-dir\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508389 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-dshm\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508389 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-home\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508666 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvm9v\" (UniqueName: \"kubernetes.io/projected/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-kube-api-access-tvm9v\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508666 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tls-certs\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508666 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tmp-dir\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508666 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-model-cache\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508862 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508804    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-home\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508862 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tmp-dir\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.508940 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.508929    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-model-cache\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.510722 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.510693    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-dshm\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.511028 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.511008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tls-certs\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.516759 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.516734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvm9v\" (UniqueName: \"kubernetes.io/projected/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-kube-api-access-tvm9v\") pod \"e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") " pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.623751 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.623709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:00:52.759754 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:52.759689    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"]
Jul 23 01:00:52.762165 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:00:52.762133    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7586fac8_35d9_4638_aaa6_7a34c6b6c1b0.slice/crio-15a935435da00d37f5460d8929d15a91bd019462148a0bb4561a60effe560264 WatchSource:0}: Error finding container 15a935435da00d37f5460d8929d15a91bd019462148a0bb4561a60effe560264: Status 404 returned error can't find the container with id 15a935435da00d37f5460d8929d15a91bd019462148a0bb4561a60effe560264
Jul 23 01:00:53.432970 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:53.432930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq" event={"ID":"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0","Type":"ContainerStarted","Data":"24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1"}
Jul 23 01:00:53.433378 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:53.432980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq" event={"ID":"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0","Type":"ContainerStarted","Data":"15a935435da00d37f5460d8929d15a91bd019462148a0bb4561a60effe560264"}
Jul 23 01:00:53.454526 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:53.454470    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq" podStartSLOduration=1.454454051 podStartE2EDuration="1.454454051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:53.452774876 +0000 UTC m=+2247.248654139" watchObservedRunningTime="2026-07-23 01:00:53.454454051 +0000 UTC m=+2247.250333301"
Jul 23 01:00:56.680572 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:00:56.680536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:01:02.624737 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:02.624693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:01:02.625167 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:02.624753    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:01:02.637333 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:02.637306    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:01:03.492730 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:03.492698    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:01:12.579330 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:12.579292    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:01:33.482691 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:33.482643    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:01:43.381303 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:43.381266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:01:46.427480 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:01:46.427447    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"]
Jul 23 01:02:01.447814 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.447749    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq" podUID="7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" containerName="main" containerID="cri-o://24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1" gracePeriod=45
Jul 23 01:02:01.706025 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.705966    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:02:01.757716 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.757684    2569 generic.go:356] "Generic (PLEG): container finished" podID="7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" containerID="24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1" exitCode=0
Jul 23 01:02:01.757716 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.757723    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"
Jul 23 01:02:01.757940 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.757761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq" event={"ID":"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0","Type":"ContainerDied","Data":"24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1"}
Jul 23 01:02:01.757940 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.757807    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq" event={"ID":"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0","Type":"ContainerDied","Data":"15a935435da00d37f5460d8929d15a91bd019462148a0bb4561a60effe560264"}
Jul 23 01:02:01.757940 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.757830    2569 scope.go:117] "RemoveContainer" containerID="24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1"
Jul 23 01:02:01.767698 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.767676    2569 scope.go:117] "RemoveContainer" containerID="24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1"
Jul 23 01:02:01.767969 ip-10-0-139-138 kubenswrapper[2569]: E0723 01:02:01.767948    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1\": container with ID starting with 24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1 not found: ID does not exist" containerID="24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1"
Jul 23 01:02:01.768036 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.767982    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1"} err="failed to get container status \"24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1\": rpc error: code = NotFound desc = could not find container \"24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1\": container with ID starting with 24768a701d87e8446ffeca216a0506751ff396437b2c92ff8f5d1dc34ea4e7b1 not found: ID does not exist"
Jul 23 01:02:01.835646 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-home\") pod \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") "
Jul 23 01:02:01.835646 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-dshm\") pod \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") "
Jul 23 01:02:01.835846 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835703    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-model-cache\") pod \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") "
Jul 23 01:02:01.835846 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835724    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tls-certs\") pod \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") "
Jul 23 01:02:01.835846 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvm9v\" (UniqueName: \"kubernetes.io/projected/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-kube-api-access-tvm9v\") pod \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") "
Jul 23 01:02:01.835846 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835800    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tmp-dir\") pod \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\" (UID: \"7586fac8-35d9-4638-aaa6-7a34c6b6c1b0\") "
Jul 23 01:02:01.836028 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.835963    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-home" (OuterVolumeSpecName: "home") pod "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" (UID: "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:02:01.836110 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.836042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" (UID: "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:02:01.836110 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.836060    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:02:01.836227 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.836205    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" (UID: "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:02:01.838118 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.838092    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" (UID: "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 01:02:01.838206 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.838093    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-kube-api-access-tvm9v" (OuterVolumeSpecName: "kube-api-access-tvm9v") pod "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" (UID: "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0"). InnerVolumeSpecName "kube-api-access-tvm9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 01:02:01.838206 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.838125    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-dshm" (OuterVolumeSpecName: "dshm") pod "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" (UID: "7586fac8-35d9-4638-aaa6-7a34c6b6c1b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:02:01.936728 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.936687    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvm9v\" (UniqueName: \"kubernetes.io/projected/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-kube-api-access-tvm9v\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:02:01.936918 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.936742    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:02:01.936918 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.936753    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:02:01.936918 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.936762    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:02:01.936918 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:01.936770    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:02:02.079951 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:02.079918    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"]
Jul 23 01:02:02.083367 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:02.083340    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0828012d/e2e-lifecycle-model-0828012d-kserve-c98bdc655-rwstq"]
Jul 23 01:02:02.746445 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:02.746414    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" path="/var/lib/kubelet/pods/7586fac8-35d9-4638-aaa6-7a34c6b6c1b0/volumes"
Jul 23 01:02:33.086792 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:33.086755    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:02:45.685908 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:02:45.685873    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:03:26.774912 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:03:26.774880    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:03:26.781715 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:03:26.781693    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 01:03:26.784312 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:03:26.784293    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:03:26.790396 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:03:26.790377    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 01:03:33.250047 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:03:33.250010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:03:49.183052 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:03:49.182967    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:04:04.092185 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.092146    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"]
Jul 23 01:04:04.092648 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.092520    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" containerName="main"
Jul 23 01:04:04.092648 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.092531    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" containerName="main"
Jul 23 01:04:04.092648 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.092585    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7586fac8-35d9-4638-aaa6-7a34c6b6c1b0" containerName="main"
Jul 23 01:04:04.095519 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.095499    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.098091 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.098057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ce8306\"/\"default-dockercfg-wbtrm\""
Jul 23 01:04:04.099100 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.099057    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ce8306\"/\"kube-root-ca.crt\""
Jul 23 01:04:04.099193 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.099170    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ce8306\"/\"openshift-service-ca.crt\""
Jul 23 01:04:04.099261 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.099198    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ce8306\"/\"test-model-ce8306-kserve-self-signed-certs\""
Jul 23 01:04:04.105153 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.105131    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"]
Jul 23 01:04:04.139112 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.139059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-dshm\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.139288 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.139121    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-home\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.139288 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.139154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-tmp-dir\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.139288 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.139275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5745bf52-1c6e-475d-9977-4f7f763857b3-tls-certs\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.139414 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.139310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwth8\" (UniqueName: \"kubernetes.io/projected/5745bf52-1c6e-475d-9977-4f7f763857b3-kube-api-access-jwth8\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.139414 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.139346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-model-cache\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240376 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5745bf52-1c6e-475d-9977-4f7f763857b3-tls-certs\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240376 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwth8\" (UniqueName: \"kubernetes.io/projected/5745bf52-1c6e-475d-9977-4f7f763857b3-kube-api-access-jwth8\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240664 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-model-cache\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240664 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-dshm\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240664 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-home\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240664 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-tmp-dir\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.240895 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-model-cache\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.241018 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.240997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-tmp-dir\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.241093 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.241028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-home\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.242753 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.242727    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-dshm\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.243047 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.243030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5745bf52-1c6e-475d-9977-4f7f763857b3-tls-certs\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.253137 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.248602    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwth8\" (UniqueName: \"kubernetes.io/projected/5745bf52-1c6e-475d-9977-4f7f763857b3-kube-api-access-jwth8\") pod \"test-model-ce8306-kserve-77f7ccf6b7-c7m7t\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") " pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.405152 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.405096    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:04.540147 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:04.540111    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"]
Jul 23 01:04:04.543058 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:04:04.543027    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5745bf52_1c6e_475d_9977_4f7f763857b3.slice/crio-0743e1ad4559173480c28ce541024f919773072a3b75b26bc82e800b90c687d8 WatchSource:0}: Error finding container 0743e1ad4559173480c28ce541024f919773072a3b75b26bc82e800b90c687d8: Status 404 returned error can't find the container with id 0743e1ad4559173480c28ce541024f919773072a3b75b26bc82e800b90c687d8
Jul 23 01:04:05.329349 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:05.329312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t" event={"ID":"5745bf52-1c6e-475d-9977-4f7f763857b3","Type":"ContainerStarted","Data":"8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f"}
Jul 23 01:04:05.329349 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:05.329352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t" event={"ID":"5745bf52-1c6e-475d-9977-4f7f763857b3","Type":"ContainerStarted","Data":"0743e1ad4559173480c28ce541024f919773072a3b75b26bc82e800b90c687d8"}
Jul 23 01:04:05.347215 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:05.347146    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t" podStartSLOduration=1.347128994 podStartE2EDuration="1.347128994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:04:05.345457338 +0000 UTC m=+2439.141336611" watchObservedRunningTime="2026-07-23 01:04:05.347128994 +0000 UTC m=+2439.143008284"
Jul 23 01:04:08.819648 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:08.819607    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:04:14.405418 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:14.405372    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:14.405812 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:14.405440    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:14.418603 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:14.418577    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:15.385456 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:15.385428    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:04:38.379711 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:38.379679    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:04:52.185127 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:04:52.185089    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:05:14.821464 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:14.821431    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"]
Jul 23 01:05:19.680816 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:19.680739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:05:29.842617 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:29.842575    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t" podUID="5745bf52-1c6e-475d-9977-4f7f763857b3" containerName="main" containerID="cri-o://8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f" gracePeriod=45
Jul 23 01:05:30.091212 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.091180    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:05:30.135208 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135172    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-home\") pod \"5745bf52-1c6e-475d-9977-4f7f763857b3\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") "
Jul 23 01:05:30.135208 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135206    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-tmp-dir\") pod \"5745bf52-1c6e-475d-9977-4f7f763857b3\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") "
Jul 23 01:05:30.135457 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135258    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-dshm\") pod \"5745bf52-1c6e-475d-9977-4f7f763857b3\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") "
Jul 23 01:05:30.135457 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135276    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwth8\" (UniqueName: \"kubernetes.io/projected/5745bf52-1c6e-475d-9977-4f7f763857b3-kube-api-access-jwth8\") pod \"5745bf52-1c6e-475d-9977-4f7f763857b3\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") "
Jul 23 01:05:30.135457 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135301    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5745bf52-1c6e-475d-9977-4f7f763857b3-tls-certs\") pod \"5745bf52-1c6e-475d-9977-4f7f763857b3\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") "
Jul 23 01:05:30.135457 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135362    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-model-cache\") pod \"5745bf52-1c6e-475d-9977-4f7f763857b3\" (UID: \"5745bf52-1c6e-475d-9977-4f7f763857b3\") "
Jul 23 01:05:30.135654 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135489    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-home" (OuterVolumeSpecName: "home") pod "5745bf52-1c6e-475d-9977-4f7f763857b3" (UID: "5745bf52-1c6e-475d-9977-4f7f763857b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:05:30.135654 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135587    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5745bf52-1c6e-475d-9977-4f7f763857b3" (UID: "5745bf52-1c6e-475d-9977-4f7f763857b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:05:30.135769 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135701    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5745bf52-1c6e-475d-9977-4f7f763857b3" (UID: "5745bf52-1c6e-475d-9977-4f7f763857b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:05:30.135844 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135824    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:05:30.135899 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135848    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:05:30.135899 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.135863    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:05:30.137644 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.137616    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-dshm" (OuterVolumeSpecName: "dshm") pod "5745bf52-1c6e-475d-9977-4f7f763857b3" (UID: "5745bf52-1c6e-475d-9977-4f7f763857b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:05:30.137644 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.137632    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5745bf52-1c6e-475d-9977-4f7f763857b3-kube-api-access-jwth8" (OuterVolumeSpecName: "kube-api-access-jwth8") pod "5745bf52-1c6e-475d-9977-4f7f763857b3" (UID: "5745bf52-1c6e-475d-9977-4f7f763857b3"). InnerVolumeSpecName "kube-api-access-jwth8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 01:05:30.137785 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.137684    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5745bf52-1c6e-475d-9977-4f7f763857b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5745bf52-1c6e-475d-9977-4f7f763857b3" (UID: "5745bf52-1c6e-475d-9977-4f7f763857b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 01:05:30.237358 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.237325    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5745bf52-1c6e-475d-9977-4f7f763857b3-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:05:30.237358 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.237351    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5745bf52-1c6e-475d-9977-4f7f763857b3-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:05:30.237358 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.237361    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwth8\" (UniqueName: \"kubernetes.io/projected/5745bf52-1c6e-475d-9977-4f7f763857b3-kube-api-access-jwth8\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:05:30.733674 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.733640    2569 generic.go:356] "Generic (PLEG): container finished" podID="5745bf52-1c6e-475d-9977-4f7f763857b3" containerID="8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f" exitCode=0
Jul 23 01:05:30.733853 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.733686    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"
Jul 23 01:05:30.733853 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.733728    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t" event={"ID":"5745bf52-1c6e-475d-9977-4f7f763857b3","Type":"ContainerDied","Data":"8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f"}
Jul 23 01:05:30.733853 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.733774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t" event={"ID":"5745bf52-1c6e-475d-9977-4f7f763857b3","Type":"ContainerDied","Data":"0743e1ad4559173480c28ce541024f919773072a3b75b26bc82e800b90c687d8"}
Jul 23 01:05:30.733853 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.733795    2569 scope.go:117] "RemoveContainer" containerID="8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f"
Jul 23 01:05:30.743483 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.743460    2569 scope.go:117] "RemoveContainer" containerID="8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f"
Jul 23 01:05:30.743796 ip-10-0-139-138 kubenswrapper[2569]: E0723 01:05:30.743776    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f\": container with ID starting with 8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f not found: ID does not exist" containerID="8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f"
Jul 23 01:05:30.743860 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.743810    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f"} err="failed to get container status \"8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f\": rpc error: code = NotFound desc = could not find container \"8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f\": container with ID starting with 8a73812c9658e78c9eff100e00b44c1c6f2f11dd4b71b41c35436dabaca8555f not found: ID does not exist"
Jul 23 01:05:30.755163 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.755136    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"]
Jul 23 01:05:30.761210 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:30.761181    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ce8306/test-model-ce8306-kserve-77f7ccf6b7-c7m7t"]
Jul 23 01:05:32.745614 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:32.745573    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5745bf52-1c6e-475d-9977-4f7f763857b3" path="/var/lib/kubelet/pods/5745bf52-1c6e-475d-9977-4f7f763857b3/volumes"
Jul 23 01:05:35.476888 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:35.476848    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:05:51.576023 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:05:51.575983    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:06:11.917576 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:06:11.917539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:06:28.285827 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:06:28.285789    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:06:46.077905 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:06:46.077867    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:07:11.782512 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:11.782423    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:07:26.788470 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:26.788439    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:07:42.634121 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.634085    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"]
Jul 23 01:07:42.634570 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.634461    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5745bf52-1c6e-475d-9977-4f7f763857b3" containerName="main"
Jul 23 01:07:42.634570 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.634474    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5745bf52-1c6e-475d-9977-4f7f763857b3" containerName="main"
Jul 23 01:07:42.634570 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.634531    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5745bf52-1c6e-475d-9977-4f7f763857b3" containerName="main"
Jul 23 01:07:42.637563 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.637544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.640334 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.640313    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"kube-root-ca.crt\""
Jul 23 01:07:42.641208 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.641168    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"default-dockercfg-t96hh\""
Jul 23 01:07:42.641325 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.641272    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"auth-test-model-7ee5a5-kserve-self-signed-certs\""
Jul 23 01:07:42.641325 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.641299    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"openshift-service-ca.crt\""
Jul 23 01:07:42.649756 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.649734    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"]
Jul 23 01:07:42.748435 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.748395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tmp-dir\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.748589 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.748450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-home\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.748589 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.748495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-dshm\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.748589 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.748560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tls-certs\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.748716 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.748633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmc26\" (UniqueName: \"kubernetes.io/projected/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-kube-api-access-mmc26\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.748716 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.748653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-model-cache\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849324 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmc26\" (UniqueName: \"kubernetes.io/projected/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-kube-api-access-mmc26\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849486 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-model-cache\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849486 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tmp-dir\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849486 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-home\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849486 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-dshm\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849486 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tls-certs\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849765 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-model-cache\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849826 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849768    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-home\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.849884 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.849835    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tmp-dir\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.851652 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.851629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-dshm\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.852060 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.852037    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tls-certs\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.857415 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.857393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmc26\" (UniqueName: \"kubernetes.io/projected/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-kube-api-access-mmc26\") pod \"auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") " pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:42.949191 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:42.949112    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:43.086436 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:43.086411    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"]
Jul 23 01:07:43.088598 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:07:43.088572    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5f8aa30_68e3_473e_be04_bc880b1a6f5a.slice/crio-8a780c666e560a67465b1df4f548bb8a3a10f79673e0df16871e791b6afe565d WatchSource:0}: Error finding container 8a780c666e560a67465b1df4f548bb8a3a10f79673e0df16871e791b6afe565d: Status 404 returned error can't find the container with id 8a780c666e560a67465b1df4f548bb8a3a10f79673e0df16871e791b6afe565d
Jul 23 01:07:43.350712 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:43.350623    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm" event={"ID":"c5f8aa30-68e3-473e-be04-bc880b1a6f5a","Type":"ContainerStarted","Data":"189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0"}
Jul 23 01:07:43.350712 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:43.350665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm" event={"ID":"c5f8aa30-68e3-473e-be04-bc880b1a6f5a","Type":"ContainerStarted","Data":"8a780c666e560a67465b1df4f548bb8a3a10f79673e0df16871e791b6afe565d"}
Jul 23 01:07:43.368822 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:43.368776    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm" podStartSLOduration=1.36876109 podStartE2EDuration="1.36876109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:07:43.367175827 +0000 UTC m=+2657.163055088" watchObservedRunningTime="2026-07-23 01:07:43.36876109 +0000 UTC m=+2657.164640341"
Jul 23 01:07:43.695834 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:43.695800    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:07:52.949433 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:52.949398    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:52.949433 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:52.949443    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:52.961754 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:52.961728    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:53.407902 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:53.407875    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:07:54.477831 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:07:54.477799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:08:12.602127 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.602093    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"]
Jul 23 01:08:12.605679 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.605656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.608694 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.608665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"kube-root-ca.crt\""
Jul 23 01:08:12.608694 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.608689    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"openshift-service-ca.crt\""
Jul 23 01:08:12.609790 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.609766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"auth-test-model-d6869a-kserve-self-signed-certs\""
Jul 23 01:08:12.609906 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.609768    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"default-dockercfg-d58v2\""
Jul 23 01:08:12.617003 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.616980    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"]
Jul 23 01:08:12.700530 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.700493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-dshm\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.700530 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.700527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-model-cache\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.700782 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.700559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-home\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.700782 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.700631    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6j2\" (UniqueName: \"kubernetes.io/projected/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-kube-api-access-wc6j2\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.700782 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.700663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tmp-dir\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.700782 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.700691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tls-certs\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801526 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6j2\" (UniqueName: \"kubernetes.io/projected/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-kube-api-access-wc6j2\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801526 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tmp-dir\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801758 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tls-certs\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801758 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-dshm\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801758 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-model-cache\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801919 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-home\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.801977 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.801952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tmp-dir\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.802034 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.802004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-model-cache\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.802124 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.802107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-home\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.803897 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.803875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-dshm\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.804055 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.804038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tls-certs\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.810574 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.810556    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6j2\" (UniqueName: \"kubernetes.io/projected/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-kube-api-access-wc6j2\") pod \"auth-test-model-d6869a-kserve-6fd8469dc5-2skbz\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") " pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:12.917010 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:12.916974    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:13.057397 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:13.057370    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"]
Jul 23 01:08:13.059318 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:08:13.059289    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fc3fa96_3ade_4b7c_89e4_c64a925bb1b7.slice/crio-c8f2549e0bf42e7a3e0bb68c09afd55af3cfee66ea104382a70471ec1148a7c4 WatchSource:0}: Error finding container c8f2549e0bf42e7a3e0bb68c09afd55af3cfee66ea104382a70471ec1148a7c4: Status 404 returned error can't find the container with id c8f2549e0bf42e7a3e0bb68c09afd55af3cfee66ea104382a70471ec1148a7c4
Jul 23 01:08:13.487376 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:13.487343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz" event={"ID":"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7","Type":"ContainerStarted","Data":"60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14"}
Jul 23 01:08:13.487641 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:13.487387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz" event={"ID":"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7","Type":"ContainerStarted","Data":"c8f2549e0bf42e7a3e0bb68c09afd55af3cfee66ea104382a70471ec1148a7c4"}
Jul 23 01:08:13.511112 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:13.511049    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz" podStartSLOduration=1.511035287 podStartE2EDuration="1.511035287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:08:13.509023589 +0000 UTC m=+2687.304902840" watchObservedRunningTime="2026-07-23 01:08:13.511035287 +0000 UTC m=+2687.306914537"
Jul 23 01:08:16.187685 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:16.187646    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:08:22.917584 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:22.917501    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:22.917584 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:22.917553    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:22.929800 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:22.929775    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:23.548303 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:23.548275    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:08:26.802984 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:26.802957    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:08:26.808879 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:26.808854    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 01:08:26.813139 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:26.813122    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:08:26.818370 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:26.818346    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 01:08:32.986676 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:32.986642    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:08:50.985188 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:08:50.985150    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:09:10.576703 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:09:10.576661    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:09:27.187833 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:09:27.187795    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:09:44.189928 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:09:44.189893    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:10:03.583176 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:03.583033    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:10:22.583166 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:22.583123    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:10:27.780888 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:27.780854    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"]
Jul 23 01:10:28.769808 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:28.769773    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"]
Jul 23 01:10:39.479127 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:39.479089    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:10:42.800570 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:42.800536    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm" podUID="c5f8aa30-68e3-473e-be04-bc880b1a6f5a" containerName="main" containerID="cri-o://189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0" gracePeriod=45
Jul 23 01:10:43.062219 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.062163    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:10:43.139554 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139513    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tls-certs\") pod \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") "
Jul 23 01:10:43.139743 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139574    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-home\") pod \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") "
Jul 23 01:10:43.139743 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139614    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tmp-dir\") pod \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") "
Jul 23 01:10:43.139743 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139637    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-dshm\") pod \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") "
Jul 23 01:10:43.139743 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139676    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmc26\" (UniqueName: \"kubernetes.io/projected/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-kube-api-access-mmc26\") pod \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") "
Jul 23 01:10:43.139743 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-model-cache\") pod \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\" (UID: \"c5f8aa30-68e3-473e-be04-bc880b1a6f5a\") "
Jul 23 01:10:43.139990 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139867    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-home" (OuterVolumeSpecName: "home") pod "c5f8aa30-68e3-473e-be04-bc880b1a6f5a" (UID: "c5f8aa30-68e3-473e-be04-bc880b1a6f5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:43.139990 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.139912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5f8aa30-68e3-473e-be04-bc880b1a6f5a" (UID: "c5f8aa30-68e3-473e-be04-bc880b1a6f5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:43.140133 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.140027    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5f8aa30-68e3-473e-be04-bc880b1a6f5a" (UID: "c5f8aa30-68e3-473e-be04-bc880b1a6f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:43.140133 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.140107    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:43.140133 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.140123    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:43.141891 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.141862    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-kube-api-access-mmc26" (OuterVolumeSpecName: "kube-api-access-mmc26") pod "c5f8aa30-68e3-473e-be04-bc880b1a6f5a" (UID: "c5f8aa30-68e3-473e-be04-bc880b1a6f5a"). InnerVolumeSpecName "kube-api-access-mmc26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 01:10:43.141891 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.141884    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5f8aa30-68e3-473e-be04-bc880b1a6f5a" (UID: "c5f8aa30-68e3-473e-be04-bc880b1a6f5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 01:10:43.141891 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.141863    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-dshm" (OuterVolumeSpecName: "dshm") pod "c5f8aa30-68e3-473e-be04-bc880b1a6f5a" (UID: "c5f8aa30-68e3-473e-be04-bc880b1a6f5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:43.210947 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.210917    2569 generic.go:356] "Generic (PLEG): container finished" podID="c5f8aa30-68e3-473e-be04-bc880b1a6f5a" containerID="189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0" exitCode=0
Jul 23 01:10:43.211137 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.210959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm" event={"ID":"c5f8aa30-68e3-473e-be04-bc880b1a6f5a","Type":"ContainerDied","Data":"189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0"}
Jul 23 01:10:43.211137 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.210970    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"
Jul 23 01:10:43.211137 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.210988    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm" event={"ID":"c5f8aa30-68e3-473e-be04-bc880b1a6f5a","Type":"ContainerDied","Data":"8a780c666e560a67465b1df4f548bb8a3a10f79673e0df16871e791b6afe565d"}
Jul 23 01:10:43.211137 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.211005    2569 scope.go:117] "RemoveContainer" containerID="189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0"
Jul 23 01:10:43.220492 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.220473    2569 scope.go:117] "RemoveContainer" containerID="189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0"
Jul 23 01:10:43.220761 ip-10-0-139-138 kubenswrapper[2569]: E0723 01:10:43.220741    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0\": container with ID starting with 189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0 not found: ID does not exist" containerID="189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0"
Jul 23 01:10:43.220826 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.220768    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0"} err="failed to get container status \"189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0\": rpc error: code = NotFound desc = could not find container \"189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0\": container with ID starting with 189dff1c78326c86f7e26bc6d0f1f735a8dc0484c2030590defc2597bac69ea0 not found: ID does not exist"
Jul 23 01:10:43.232795 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.232769    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"]
Jul 23 01:10:43.238300 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.238278    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/auth-test-model-7ee5a5-kserve-59f9cd74d7-fw7cm"]
Jul 23 01:10:43.240813 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.240797    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:43.240869 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.240819    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:43.240869 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.240831    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmc26\" (UniqueName: \"kubernetes.io/projected/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-kube-api-access-mmc26\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:43.240869 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.240840    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8aa30-68e3-473e-be04-bc880b1a6f5a-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:43.790633 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:43.790569    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz" podUID="6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" containerName="main" containerID="cri-o://60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14" gracePeriod=45
Jul 23 01:10:44.046546 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.046481    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:10:44.146857 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.146827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-dshm\") pod \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") "
Jul 23 01:10:44.147031 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.146915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc6j2\" (UniqueName: \"kubernetes.io/projected/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-kube-api-access-wc6j2\") pod \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") "
Jul 23 01:10:44.147031 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.146944    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-model-cache\") pod \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") "
Jul 23 01:10:44.147031 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.146963    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tmp-dir\") pod \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") "
Jul 23 01:10:44.147031 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.146993    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-home\") pod \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") "
Jul 23 01:10:44.147266 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147132    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tls-certs\") pod \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\" (UID: \"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7\") "
Jul 23 01:10:44.147320 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147266    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" (UID: "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:44.147320 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147278    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-home" (OuterVolumeSpecName: "home") pod "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" (UID: "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:44.147320 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147301    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" (UID: "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:44.147492 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147474    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:44.147544 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147496    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:44.147544 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.147510    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:44.149172 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.149149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-dshm" (OuterVolumeSpecName: "dshm") pod "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" (UID: "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:10:44.149266 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.149235    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" (UID: "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 01:10:44.149326 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.149307    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-kube-api-access-wc6j2" (OuterVolumeSpecName: "kube-api-access-wc6j2") pod "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" (UID: "6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7"). InnerVolumeSpecName "kube-api-access-wc6j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 01:10:44.217594 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.217559    2569 generic.go:356] "Generic (PLEG): container finished" podID="6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" containerID="60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14" exitCode=0
Jul 23 01:10:44.217736 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.217610    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"
Jul 23 01:10:44.217736 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.217642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz" event={"ID":"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7","Type":"ContainerDied","Data":"60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14"}
Jul 23 01:10:44.217736 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.217682    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz" event={"ID":"6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7","Type":"ContainerDied","Data":"c8f2549e0bf42e7a3e0bb68c09afd55af3cfee66ea104382a70471ec1148a7c4"}
Jul 23 01:10:44.217736 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.217698    2569 scope.go:117] "RemoveContainer" containerID="60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14"
Jul 23 01:10:44.226285 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.226263    2569 scope.go:117] "RemoveContainer" containerID="60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14"
Jul 23 01:10:44.226627 ip-10-0-139-138 kubenswrapper[2569]: E0723 01:10:44.226607    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14\": container with ID starting with 60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14 not found: ID does not exist" containerID="60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14"
Jul 23 01:10:44.226695 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.226634    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14"} err="failed to get container status \"60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14\": rpc error: code = NotFound desc = could not find container \"60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14\": container with ID starting with 60956e65571301135195e84d489e86886a17ec4023c3a4b8f29e9c2714029e14 not found: ID does not exist"
Jul 23 01:10:44.239710 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.239683    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"]
Jul 23 01:10:44.244879 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.244860    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/auth-test-model-d6869a-kserve-6fd8469dc5-2skbz"]
Jul 23 01:10:44.248497 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.248482    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:44.248558 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.248500    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:44.248558 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.248509    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc6j2\" (UniqueName: \"kubernetes.io/projected/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7-kube-api-access-wc6j2\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:10:44.745349 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.745319    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" path="/var/lib/kubelet/pods/6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7/volumes"
Jul 23 01:10:44.745770 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:44.745756    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f8aa30-68e3-473e-be04-bc880b1a6f5a" path="/var/lib/kubelet/pods/c5f8aa30-68e3-473e-be04-bc880b1a6f5a/volumes"
Jul 23 01:10:56.889681 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.889651    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"]
Jul 23 01:10:56.890134 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.889997    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f8aa30-68e3-473e-be04-bc880b1a6f5a" containerName="main"
Jul 23 01:10:56.890134 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.890009    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f8aa30-68e3-473e-be04-bc880b1a6f5a" containerName="main"
Jul 23 01:10:56.890134 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.890026    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" containerName="main"
Jul 23 01:10:56.890134 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.890032    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" containerName="main"
Jul 23 01:10:56.890134 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.890100    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc3fa96-3ade-4b7c-89e4-c64a925bb1b7" containerName="main"
Jul 23 01:10:56.890134 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.890112    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f8aa30-68e3-473e-be04-bc880b1a6f5a" containerName="main"
Jul 23 01:10:56.894446 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.894426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:56.897950 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.897928    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"sub-test-model-d6869a-kserve-self-signed-certs\""
Jul 23 01:10:56.898089 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.897928    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"default-dockercfg-d58v2\""
Jul 23 01:10:56.898089 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.897934    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"kube-root-ca.crt\""
Jul 23 01:10:56.898089 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.897934    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"openshift-service-ca.crt\""
Jul 23 01:10:56.902926 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.902893    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"]
Jul 23 01:10:56.955917 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.955886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-home\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:56.956059 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.955924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-model-cache\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:56.956059 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.955944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-dshm\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:56.956059 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.955965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmpg\" (UniqueName: \"kubernetes.io/projected/d5f995d2-7721-4721-be7b-038235980e52-kube-api-access-wqmpg\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:56.956059 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.955995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f995d2-7721-4721-be7b-038235980e52-tls-certs\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:56.956059 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:56.956012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-tmp-dir\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.057441 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.057403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmpg\" (UniqueName: \"kubernetes.io/projected/d5f995d2-7721-4721-be7b-038235980e52-kube-api-access-wqmpg\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.057627 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.057452    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f995d2-7721-4721-be7b-038235980e52-tls-certs\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.057627 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.057481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-tmp-dir\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.057627 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.057608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-home\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.057814 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.057648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-model-cache\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.057814 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.057676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-dshm\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.058112 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.058062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-model-cache\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.058209 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.058135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-tmp-dir\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.058261 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.058233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-home\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.060024 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.059992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-dshm\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.060147 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.060060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f995d2-7721-4721-be7b-038235980e52-tls-certs\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.066203 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.066185    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmpg\" (UniqueName: \"kubernetes.io/projected/d5f995d2-7721-4721-be7b-038235980e52-kube-api-access-wqmpg\") pod \"sub-test-model-d6869a-kserve-8df94c6cf-7mlzp\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") " pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.204013 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.203945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:10:57.333631 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:57.333604    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"]
Jul 23 01:10:57.336254 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:10:57.336223    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5f995d2_7721_4721_be7b_038235980e52.slice/crio-da29dbef7fc0cfae1540d0c19ef584cb4e465272a110c1c08b5ac1dbbe67bb5b WatchSource:0}: Error finding container da29dbef7fc0cfae1540d0c19ef584cb4e465272a110c1c08b5ac1dbbe67bb5b: Status 404 returned error can't find the container with id da29dbef7fc0cfae1540d0c19ef584cb4e465272a110c1c08b5ac1dbbe67bb5b
Jul 23 01:10:58.286154 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:58.286120    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp" event={"ID":"d5f995d2-7721-4721-be7b-038235980e52","Type":"ContainerStarted","Data":"231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e"}
Jul 23 01:10:58.286521 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:58.286161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp" event={"ID":"d5f995d2-7721-4721-be7b-038235980e52","Type":"ContainerStarted","Data":"da29dbef7fc0cfae1540d0c19ef584cb4e465272a110c1c08b5ac1dbbe67bb5b"}
Jul 23 01:10:58.303760 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:10:58.303715    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp" podStartSLOduration=2.303698524 podStartE2EDuration="2.303698524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:10:58.301771319 +0000 UTC m=+2852.097650580" watchObservedRunningTime="2026-07-23 01:10:58.303698524 +0000 UTC m=+2852.099577779"
Jul 23 01:11:00.581737 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:00.581696    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:11:07.204519 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:07.204481    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:11:07.204966 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:07.204553    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:11:07.216949 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:07.216915    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:11:07.342737 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:07.342704    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:11:22.996855 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:22.996778    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:11:36.695932 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:36.695894    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:11:44.496230 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:44.496196    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"]
Jul 23 01:11:51.387429 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.387396    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"]
Jul 23 01:11:51.391449 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.391427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.393899 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.393871    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"kube-root-ca.crt\""
Jul 23 01:11:51.394020 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.393873    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"default-dockercfg-t96hh\""
Jul 23 01:11:51.394748 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.394728    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"openshift-service-ca.crt\""
Jul 23 01:11:51.394850 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.394830    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"rate-test-model-7ee5a5-kserve-self-signed-certs\""
Jul 23 01:11:51.398447 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.398425    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"]
Jul 23 01:11:51.423655 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.423632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hd6k\" (UniqueName: \"kubernetes.io/projected/e7c44c54-124e-4625-8138-cdc634237ee7-kube-api-access-8hd6k\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.423755 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.423663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-home\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.423755 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.423688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-dshm\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.423755 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.423703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-model-cache\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.423888 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.423792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7c44c54-124e-4625-8138-cdc634237ee7-tls-certs\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.423888 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.423829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-tmp-dir\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525310 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525277    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hd6k\" (UniqueName: \"kubernetes.io/projected/e7c44c54-124e-4625-8138-cdc634237ee7-kube-api-access-8hd6k\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525451 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-home\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525451 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-dshm\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525567 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-model-cache\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525623 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7c44c54-124e-4625-8138-cdc634237ee7-tls-certs\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525675 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-tmp-dir\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525675 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-home\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525816 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525791    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-model-cache\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.525950 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.525931    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-tmp-dir\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.527484 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.527468    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-dshm\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.527891 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.527875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7c44c54-124e-4625-8138-cdc634237ee7-tls-certs\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.533947 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.533925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hd6k\" (UniqueName: \"kubernetes.io/projected/e7c44c54-124e-4625-8138-cdc634237ee7-kube-api-access-8hd6k\") pod \"rate-test-model-7ee5a5-kserve-7557d86795-mbdcl\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") " pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.701850 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.701753    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:11:51.837214 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:51.837188    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"]
Jul 23 01:11:51.839102 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:11:51.839062    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c44c54_124e_4625_8138_cdc634237ee7.slice/crio-744e7558f9a8d57083f913bfd866a902523a5fad0227bd01e962bad711e1cede WatchSource:0}: Error finding container 744e7558f9a8d57083f913bfd866a902523a5fad0227bd01e962bad711e1cede: Status 404 returned error can't find the container with id 744e7558f9a8d57083f913bfd866a902523a5fad0227bd01e962bad711e1cede
Jul 23 01:11:52.539827 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:52.539791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl" event={"ID":"e7c44c54-124e-4625-8138-cdc634237ee7","Type":"ContainerStarted","Data":"72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac"}
Jul 23 01:11:52.540202 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:52.539838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl" event={"ID":"e7c44c54-124e-4625-8138-cdc634237ee7","Type":"ContainerStarted","Data":"744e7558f9a8d57083f913bfd866a902523a5fad0227bd01e962bad711e1cede"}
Jul 23 01:11:52.557460 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:52.557415    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl" podStartSLOduration=1.557398273 podStartE2EDuration="1.557398273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:11:52.556034702 +0000 UTC m=+2906.351913954" watchObservedRunningTime="2026-07-23 01:11:52.557398273 +0000 UTC m=+2906.353277524"
Jul 23 01:11:55.484970 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:55.484919    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:11:59.516649 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.516590    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp" podUID="d5f995d2-7721-4721-be7b-038235980e52" containerName="main" containerID="cri-o://231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e" gracePeriod=45
Jul 23 01:11:59.771321 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.771267    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:11:59.896524 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896490    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-dshm\") pod \"d5f995d2-7721-4721-be7b-038235980e52\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") "
Jul 23 01:11:59.896679 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896530    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f995d2-7721-4721-be7b-038235980e52-tls-certs\") pod \"d5f995d2-7721-4721-be7b-038235980e52\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") "
Jul 23 01:11:59.896679 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-home\") pod \"d5f995d2-7721-4721-be7b-038235980e52\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") "
Jul 23 01:11:59.896679 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896615    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-tmp-dir\") pod \"d5f995d2-7721-4721-be7b-038235980e52\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") "
Jul 23 01:11:59.896679 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqmpg\" (UniqueName: \"kubernetes.io/projected/d5f995d2-7721-4721-be7b-038235980e52-kube-api-access-wqmpg\") pod \"d5f995d2-7721-4721-be7b-038235980e52\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") "
Jul 23 01:11:59.896893 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896698    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-model-cache\") pod \"d5f995d2-7721-4721-be7b-038235980e52\" (UID: \"d5f995d2-7721-4721-be7b-038235980e52\") "
Jul 23 01:11:59.896961 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896891    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-home" (OuterVolumeSpecName: "home") pod "d5f995d2-7721-4721-be7b-038235980e52" (UID: "d5f995d2-7721-4721-be7b-038235980e52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:11:59.896961 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.896919    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5f995d2-7721-4721-be7b-038235980e52" (UID: "d5f995d2-7721-4721-be7b-038235980e52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:11:59.897052 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.897017    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:11:59.897052 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.897017    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5f995d2-7721-4721-be7b-038235980e52" (UID: "d5f995d2-7721-4721-be7b-038235980e52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:11:59.897052 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.897036    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:11:59.898630 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.898606    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-dshm" (OuterVolumeSpecName: "dshm") pod "d5f995d2-7721-4721-be7b-038235980e52" (UID: "d5f995d2-7721-4721-be7b-038235980e52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:11:59.898630 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.898623    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f995d2-7721-4721-be7b-038235980e52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5f995d2-7721-4721-be7b-038235980e52" (UID: "d5f995d2-7721-4721-be7b-038235980e52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 01:11:59.898774 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.898740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f995d2-7721-4721-be7b-038235980e52-kube-api-access-wqmpg" (OuterVolumeSpecName: "kube-api-access-wqmpg") pod "d5f995d2-7721-4721-be7b-038235980e52" (UID: "d5f995d2-7721-4721-be7b-038235980e52"). InnerVolumeSpecName "kube-api-access-wqmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 01:11:59.997698 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.997660    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqmpg\" (UniqueName: \"kubernetes.io/projected/d5f995d2-7721-4721-be7b-038235980e52-kube-api-access-wqmpg\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:11:59.997698 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.997695    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:11:59.997868 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.997706    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f995d2-7721-4721-be7b-038235980e52-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:11:59.997868 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:11:59.997714    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f995d2-7721-4721-be7b-038235980e52-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:12:00.575664 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.575631    2569 generic.go:356] "Generic (PLEG): container finished" podID="d5f995d2-7721-4721-be7b-038235980e52" containerID="231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e" exitCode=0
Jul 23 01:12:00.576082 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.575678    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"
Jul 23 01:12:00.576082 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.575706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp" event={"ID":"d5f995d2-7721-4721-be7b-038235980e52","Type":"ContainerDied","Data":"231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e"}
Jul 23 01:12:00.576082 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.575738    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp" event={"ID":"d5f995d2-7721-4721-be7b-038235980e52","Type":"ContainerDied","Data":"da29dbef7fc0cfae1540d0c19ef584cb4e465272a110c1c08b5ac1dbbe67bb5b"}
Jul 23 01:12:00.576082 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.575754    2569 scope.go:117] "RemoveContainer" containerID="231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e"
Jul 23 01:12:00.585114 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.585099    2569 scope.go:117] "RemoveContainer" containerID="231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e"
Jul 23 01:12:00.585404 ip-10-0-139-138 kubenswrapper[2569]: E0723 01:12:00.585385    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e\": container with ID starting with 231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e not found: ID does not exist" containerID="231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e"
Jul 23 01:12:00.585462 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.585413    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e"} err="failed to get container status \"231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e\": rpc error: code = NotFound desc = could not find container \"231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e\": container with ID starting with 231d3dee189aa81772a51d3ff27923730a9ec3ebd0cf05cfbcfafd2b73f85b4e not found: ID does not exist"
Jul 23 01:12:00.596986 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.596948    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"]
Jul 23 01:12:00.600351 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.600332    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/sub-test-model-d6869a-kserve-8df94c6cf-7mlzp"]
Jul 23 01:12:00.745901 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:00.745869    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f995d2-7721-4721-be7b-038235980e52" path="/var/lib/kubelet/pods/d5f995d2-7721-4721-be7b-038235980e52/volumes"
Jul 23 01:12:01.702017 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:01.701981    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:12:01.702478 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:01.702031    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:12:01.714479 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:01.714449    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:12:02.596681 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:02.596652    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:12:12.681956 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:12.681911    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:12:26.886483 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:26.886452    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:12:40.682165 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:40.682129    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:12:48.435010 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:48.434977    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"]
Jul 23 01:12:55.489198 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:12:55.489164    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:13:03.455168 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.455118    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl" podUID="e7c44c54-124e-4625-8138-cdc634237ee7" containerName="main" containerID="cri-o://72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac" gracePeriod=45
Jul 23 01:13:03.709833 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.709777    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:13:03.860990 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.860944    2569 generic.go:356] "Generic (PLEG): container finished" podID="e7c44c54-124e-4625-8138-cdc634237ee7" containerID="72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac" exitCode=0
Jul 23 01:13:03.861204 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.860997    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"
Jul 23 01:13:03.861204 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.861001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl" event={"ID":"e7c44c54-124e-4625-8138-cdc634237ee7","Type":"ContainerDied","Data":"72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac"}
Jul 23 01:13:03.861204 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.861040    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl" event={"ID":"e7c44c54-124e-4625-8138-cdc634237ee7","Type":"ContainerDied","Data":"744e7558f9a8d57083f913bfd866a902523a5fad0227bd01e962bad711e1cede"}
Jul 23 01:13:03.861204 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.861064    2569 scope.go:117] "RemoveContainer" containerID="72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac"
Jul 23 01:13:03.864432 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864412    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-tmp-dir\") pod \"e7c44c54-124e-4625-8138-cdc634237ee7\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") "
Jul 23 01:13:03.864568 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864443    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-dshm\") pod \"e7c44c54-124e-4625-8138-cdc634237ee7\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") "
Jul 23 01:13:03.864568 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864474    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hd6k\" (UniqueName: \"kubernetes.io/projected/e7c44c54-124e-4625-8138-cdc634237ee7-kube-api-access-8hd6k\") pod \"e7c44c54-124e-4625-8138-cdc634237ee7\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") "
Jul 23 01:13:03.864568 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864497    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-home\") pod \"e7c44c54-124e-4625-8138-cdc634237ee7\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") "
Jul 23 01:13:03.864568 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864528    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-model-cache\") pod \"e7c44c54-124e-4625-8138-cdc634237ee7\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") "
Jul 23 01:13:03.864568 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864544    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7c44c54-124e-4625-8138-cdc634237ee7-tls-certs\") pod \"e7c44c54-124e-4625-8138-cdc634237ee7\" (UID: \"e7c44c54-124e-4625-8138-cdc634237ee7\") "
Jul 23 01:13:03.864890 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864866    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7c44c54-124e-4625-8138-cdc634237ee7" (UID: "e7c44c54-124e-4625-8138-cdc634237ee7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:13:03.864984 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.864969    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-home" (OuterVolumeSpecName: "home") pod "e7c44c54-124e-4625-8138-cdc634237ee7" (UID: "e7c44c54-124e-4625-8138-cdc634237ee7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:13:03.865048 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.865031    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7c44c54-124e-4625-8138-cdc634237ee7" (UID: "e7c44c54-124e-4625-8138-cdc634237ee7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:13:03.866586 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.866561    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-dshm" (OuterVolumeSpecName: "dshm") pod "e7c44c54-124e-4625-8138-cdc634237ee7" (UID: "e7c44c54-124e-4625-8138-cdc634237ee7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 01:13:03.866821 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.866804    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c44c54-124e-4625-8138-cdc634237ee7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7c44c54-124e-4625-8138-cdc634237ee7" (UID: "e7c44c54-124e-4625-8138-cdc634237ee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 01:13:03.866918 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.866897    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c44c54-124e-4625-8138-cdc634237ee7-kube-api-access-8hd6k" (OuterVolumeSpecName: "kube-api-access-8hd6k") pod "e7c44c54-124e-4625-8138-cdc634237ee7" (UID: "e7c44c54-124e-4625-8138-cdc634237ee7"). InnerVolumeSpecName "kube-api-access-8hd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 01:13:03.870931 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.870911    2569 scope.go:117] "RemoveContainer" containerID="72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac"
Jul 23 01:13:03.871225 ip-10-0-139-138 kubenswrapper[2569]: E0723 01:13:03.871205    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac\": container with ID starting with 72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac not found: ID does not exist" containerID="72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac"
Jul 23 01:13:03.871285 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.871230    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac"} err="failed to get container status \"72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac\": rpc error: code = NotFound desc = could not find container \"72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac\": container with ID starting with 72c6f06705c09fcca398a2c94782f93015267849aa972f225071946dddf1e4ac not found: ID does not exist"
Jul 23 01:13:03.966095 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.966006    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hd6k\" (UniqueName: \"kubernetes.io/projected/e7c44c54-124e-4625-8138-cdc634237ee7-kube-api-access-8hd6k\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:13:03.966095 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.966029    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-home\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:13:03.966095 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.966040    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-model-cache\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:13:03.966095 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.966048    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7c44c54-124e-4625-8138-cdc634237ee7-tls-certs\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:13:03.966095 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.966055    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-tmp-dir\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:13:03.966095 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:03.966062    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7c44c54-124e-4625-8138-cdc634237ee7-dshm\") on node \"ip-10-0-139-138.ec2.internal\" DevicePath \"\""
Jul 23 01:13:04.182760 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:04.182724    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"]
Jul 23 01:13:04.186795 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:04.186766    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/rate-test-model-7ee5a5-kserve-7557d86795-mbdcl"]
Jul 23 01:13:04.746244 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:04.746200    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c44c54-124e-4625-8138-cdc634237ee7" path="/var/lib/kubelet/pods/e7c44c54-124e-4625-8138-cdc634237ee7/volumes"
Jul 23 01:13:22.486290 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:22.486250    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:13:26.829433 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:26.829402    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:13:26.835328 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:26.835295    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 01:13:26.840283 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:26.840261    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:13:26.846417 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:26.846399    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nmfpw_280511d0-25a2-4136-8252-1b3ece74cc94/ovn-acl-logging/0.log"
Jul 23 01:13:43.784650 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:43.784613    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6tpst"]
Jul 23 01:13:59.100034 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:59.099994    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9f7m6_90cf7994-066e-490a-b2d3-61a861219dce/manager/0.log"
Jul 23 01:13:59.467530 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:59.467502    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-fbzg8_e8407a8e-461f-48d0-977a-5fea3707098e/copy-manifests/0.log"
Jul 23 01:13:59.484963 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:13:59.484920    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-fbzg8_e8407a8e-461f-48d0-977a-5fea3707098e/manager/0.log"
Jul 23 01:14:01.734124 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:01.734085    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6lksz_22c81994-2110-4398-9d00-bd1a0b714d3b/manager/0.log"
Jul 23 01:14:01.981226 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:01.981195    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6snll_9f41e866-7519-434d-b271-ed797e19a27c/kuadrant-console-plugin/0.log"
Jul 23 01:14:02.126084 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:02.126036    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7bbh2_100ba5e4-0d78-4b9c-b1e0-7b9762d87e36/registry-server/0.log"
Jul 23 01:14:02.381098 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:02.380981    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6tpst_20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e/limitador/0.log"
Jul 23 01:14:02.511093 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:02.511031    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-fc7ts_e4125696-b354-4636-a0e6-dcf25658df56/manager/0.log"
Jul 23 01:14:03.739563 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:03.739531    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79b76659f8-mkbk7_5a1dde95-970c-4e29-a9d4-8acc92d30839/router/0.log"
Jul 23 01:14:04.333323 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:04.333292    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6n22g_373220b2-7e47-4921-8d87-906ff774597b/main/0.log"
Jul 23 01:14:04.452399 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:04.452367    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6fpw7_9ff7b2b9-c65b-443a-a179-c0baf8d617c6/main/0.log"
Jul 23 01:14:04.577917 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:04.577884    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-n9cdn_42b4bdd5-b5aa-4793-bbad-84812e79f96c/main/0.log"
Jul 23 01:14:04.705957 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:04.705925    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cmcr7_7ba09c19-0eb1-4b30-a842-6363172d2924/main/0.log"
Jul 23 01:14:13.911039 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:13.911003    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bvppm_6b914057-5d8f-4393-93d5-389ab25f66ad/global-pull-secret-syncer/0.log"
Jul 23 01:14:14.090990 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:14.090960    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jbdtg_a9691368-fa76-4ebe-8ad7-5b79e74bf346/konnectivity-agent/0.log"
Jul 23 01:14:14.202894 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:14.202804    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-138.ec2.internal_6cc72966d35a3fd5b5406ca4a255a917/haproxy/0.log"
Jul 23 01:14:19.435738 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:19.435657    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6lksz_22c81994-2110-4398-9d00-bd1a0b714d3b/manager/0.log"
Jul 23 01:14:19.482459 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:19.482438    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6snll_9f41e866-7519-434d-b271-ed797e19a27c/kuadrant-console-plugin/0.log"
Jul 23 01:14:19.534078 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:19.534050    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7bbh2_100ba5e4-0d78-4b9c-b1e0-7b9762d87e36/registry-server/0.log"
Jul 23 01:14:19.679870 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:19.679836    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6tpst_20d8b0f3-dcb4-4b7b-b85f-2a69b9c94f1e/limitador/0.log"
Jul 23 01:14:19.779102 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:19.779059    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-fc7ts_e4125696-b354-4636-a0e6-dcf25658df56/manager/0.log"
Jul 23 01:14:21.675084 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:21.674976    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-8hq7b_713f466d-e0af-4fd2-9fa2-1af6723e63b5/cluster-monitoring-operator/0.log"
Jul 23 01:14:21.822086 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:21.822047    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-ttw4p_e48e01a8-d3ce-4a5e-9fc2-c57221f485aa/monitoring-plugin/0.log"
Jul 23 01:14:22.028103 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.027995    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8rhl_fc46e7dd-a12b-43af-9f63-2159288c8196/node-exporter/0.log"
Jul 23 01:14:22.048770 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.048749    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8rhl_fc46e7dd-a12b-43af-9f63-2159288c8196/kube-rbac-proxy/0.log"
Jul 23 01:14:22.086857 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.086837    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8rhl_fc46e7dd-a12b-43af-9f63-2159288c8196/init-textfile/0.log"
Jul 23 01:14:22.287525 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.287444    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/prometheus/0.log"
Jul 23 01:14:22.311886 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.311853    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/config-reloader/0.log"
Jul 23 01:14:22.334562 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.334539    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/thanos-sidecar/0.log"
Jul 23 01:14:22.355364 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.355341    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/kube-rbac-proxy-web/0.log"
Jul 23 01:14:22.379321 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.379296    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/kube-rbac-proxy/0.log"
Jul 23 01:14:22.401323 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.401299    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/kube-rbac-proxy-thanos/0.log"
Jul 23 01:14:22.422966 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.422947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a043a4fe-b025-429e-bcd0-aa66cb4646f8/init-config-reloader/0.log"
Jul 23 01:14:22.457567 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.457546    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-p57gz_8faf3f05-9f00-451a-934c-2a77ea23f59a/prometheus-operator/0.log"
Jul 23 01:14:22.474991 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.474971    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-p57gz_8faf3f05-9f00-451a-934c-2a77ea23f59a/kube-rbac-proxy/0.log"
Jul 23 01:14:22.498519 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:22.498502    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-sg8nb_c234ccd0-d9e4-4757-b723-dd902958118e/prometheus-operator-admission-webhook/0.log"
Jul 23 01:14:23.819491 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819458    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"]
Jul 23 01:14:23.819924 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819825    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f995d2-7721-4721-be7b-038235980e52" containerName="main"
Jul 23 01:14:23.819924 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819836    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f995d2-7721-4721-be7b-038235980e52" containerName="main"
Jul 23 01:14:23.819924 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819854    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c44c54-124e-4625-8138-cdc634237ee7" containerName="main"
Jul 23 01:14:23.819924 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819860    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c44c54-124e-4625-8138-cdc634237ee7" containerName="main"
Jul 23 01:14:23.819924 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819920    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c44c54-124e-4625-8138-cdc634237ee7" containerName="main"
Jul 23 01:14:23.819924 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.819928    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f995d2-7721-4721-be7b-038235980e52" containerName="main"
Jul 23 01:14:23.823182 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.823160    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.825806 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.825789    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpcmt\"/\"openshift-service-ca.crt\""
Jul 23 01:14:23.826526 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.826509    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tpcmt\"/\"default-dockercfg-j8gvw\""
Jul 23 01:14:23.826578 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.826516    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpcmt\"/\"kube-root-ca.crt\""
Jul 23 01:14:23.832326 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.832308    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"]
Jul 23 01:14:23.841853 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.841825    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-sys\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.841948 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.841892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-lib-modules\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.841948 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.841926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-proc\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.842023 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.842006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-podres\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.842077 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.842030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5h96\" (UniqueName: \"kubernetes.io/projected/4a04d91d-8445-4954-89d6-d477c4ddedeb-kube-api-access-w5h96\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943412 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-podres\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943412 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5h96\" (UniqueName: \"kubernetes.io/projected/4a04d91d-8445-4954-89d6-d477c4ddedeb-kube-api-access-w5h96\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943653 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-sys\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943653 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-lib-modules\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943653 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-podres\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943653 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-proc\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943653 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-sys\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943653 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943638    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-proc\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.943850 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.943643    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a04d91d-8445-4954-89d6-d477c4ddedeb-lib-modules\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:23.951544 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:23.951520    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5h96\" (UniqueName: \"kubernetes.io/projected/4a04d91d-8445-4954-89d6-d477c4ddedeb-kube-api-access-w5h96\") pod \"perf-node-gather-daemonset-k794s\" (UID: \"4a04d91d-8445-4954-89d6-d477c4ddedeb\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:24.133313 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:24.133273    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:24.263828 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:24.263764    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"]
Jul 23 01:14:24.266253 ip-10-0-139-138 kubenswrapper[2569]: W0723 01:14:24.266220    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a04d91d_8445_4954_89d6_d477c4ddedeb.slice/crio-8b4b4a36996e310dab20f6a2cfa0102f0b1dca4a844997cf64ab3d8f30b61a2e WatchSource:0}: Error finding container 8b4b4a36996e310dab20f6a2cfa0102f0b1dca4a844997cf64ab3d8f30b61a2e: Status 404 returned error can't find the container with id 8b4b4a36996e310dab20f6a2cfa0102f0b1dca4a844997cf64ab3d8f30b61a2e
Jul 23 01:14:24.560674 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:24.560585    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/2.log"
Jul 23 01:14:24.568619 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:24.568598    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-872t6_d578e09f-5449-4d06-a9fb-a12e2247bfa9/console-operator/3.log"
Jul 23 01:14:25.237145 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:25.237107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s" event={"ID":"4a04d91d-8445-4954-89d6-d477c4ddedeb","Type":"ContainerStarted","Data":"8d5c6d5ea72b45b3653981fd9794b77c336b13152f9ea811d83f49cb2f781f8a"}
Jul 23 01:14:25.237502 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:25.237161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s" event={"ID":"4a04d91d-8445-4954-89d6-d477c4ddedeb","Type":"ContainerStarted","Data":"8b4b4a36996e310dab20f6a2cfa0102f0b1dca4a844997cf64ab3d8f30b61a2e"}
Jul 23 01:14:25.237502 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:25.237231    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:25.253636 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:25.253590    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s" podStartSLOduration=2.253575926 podStartE2EDuration="2.253575926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:14:25.251471338 +0000 UTC m=+3059.047350586" watchObservedRunningTime="2026-07-23 01:14:25.253575926 +0000 UTC m=+3059.049455203"
Jul 23 01:14:26.709388 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:26.709358    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-678wz_654828e6-8725-4ac8-98c7-709acd8b391f/dns/0.log"
Jul 23 01:14:26.729564 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:26.729540    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-678wz_654828e6-8725-4ac8-98c7-709acd8b391f/kube-rbac-proxy/0.log"
Jul 23 01:14:26.828520 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:26.828496    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dl9s5_2f49d87a-dd38-40f7-9d59-526873d06917/dns-node-resolver/0.log"
Jul 23 01:14:27.409603 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:27.409567    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-797b774bfc-2dfzh_7ec38751-2309-47af-877f-534965a54bdd/registry/0.log"
Jul 23 01:14:27.430662 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:27.430634    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6tccw_aa1702e8-c63c-4224-b143-454913151172/node-ca/0.log"
Jul 23 01:14:28.708421 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:28.708385    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79b76659f8-mkbk7_5a1dde95-970c-4e29-a9d4-8acc92d30839/router/0.log"
Jul 23 01:14:29.394106 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:29.394059    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mv4sx_74a905fb-0c3a-494c-a0e8-fa411db6f000/serve-healthcheck-canary/0.log"
Jul 23 01:14:29.909621 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:29.909589    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wzgt_0d92b989-6efc-4762-8189-133049200055/kube-rbac-proxy/0.log"
Jul 23 01:14:29.929433 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:29.929409    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wzgt_0d92b989-6efc-4762-8189-133049200055/exporter/0.log"
Jul 23 01:14:29.949669 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:29.949645    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4wzgt_0d92b989-6efc-4762-8189-133049200055/extractor/0.log"
Jul 23 01:14:31.251376 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:31.251348    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-k794s"
Jul 23 01:14:32.623361 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:32.623328    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9f7m6_90cf7994-066e-490a-b2d3-61a861219dce/manager/0.log"
Jul 23 01:14:32.868491 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:32.868460    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-fbzg8_e8407a8e-461f-48d0-977a-5fea3707098e/manager/0.log"
Jul 23 01:14:32.886294 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:32.886268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-fbzg8_e8407a8e-461f-48d0-977a-5fea3707098e/copy-manifests/0.log"
Jul 23 01:14:34.265307 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:34.265277    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85b95d447c-x2l4h_6db51faf-0d83-4606-844e-b5589bc8475a/manager/0.log"
Jul 23 01:14:34.690920 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:34.690873    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79b76659f8-mkbk7_5a1dde95-970c-4e29-a9d4-8acc92d30839/router/0.log"
Jul 23 01:14:36.163946 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:36.163916    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79b76659f8-mkbk7_5a1dde95-970c-4e29-a9d4-8acc92d30839/router/0.log"
Jul 23 01:14:39.655003 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:39.654971    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2t2vd_2d7c132e-d75d-4a98-8e1b-3209d7d998f7/kube-storage-version-migrator-operator/1.log"
Jul 23 01:14:39.656909 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:39.656886    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2t2vd_2d7c132e-d75d-4a98-8e1b-3209d7d998f7/kube-storage-version-migrator-operator/0.log"
Jul 23 01:14:40.584701 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.584669    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-29gkg_7ef7466d-00eb-4a44-972c-a48ac18eecf5/kube-multus/0.log"
Jul 23 01:14:40.605459 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.605426    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/kube-multus-additional-cni-plugins/0.log"
Jul 23 01:14:40.648484 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.648457    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/egress-router-binary-copy/0.log"
Jul 23 01:14:40.673799 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.673770    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/cni-plugins/0.log"
Jul 23 01:14:40.692605 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.692580    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/bond-cni-plugin/0.log"
Jul 23 01:14:40.712950 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.712900    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/routeoverride-cni/0.log"
Jul 23 01:14:40.732257 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.732239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/whereabouts-cni-bincopy/0.log"
Jul 23 01:14:40.750996 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:40.750978    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnx8_aee7a54d-abce-4798-9b79-bd89f42aae36/whereabouts-cni/0.log"
Jul 23 01:14:41.236913 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:41.236885    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p9dgl_d0d5d50e-1891-4bba-bf96-30fff80caffd/network-metrics-daemon/0.log"
Jul 23 01:14:41.254852 ip-10-0-139-138 kubenswrapper[2569]: I0723 01:14:41.254824    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p9dgl_d0d5d50e-1891-4bba-bf96-30fff80caffd/kube-rbac-proxy/0.log"