Jul 09 16:48:43.905316 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 16:48:43.905328 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 16:48:43.905335 ip-10-0-128-131 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 16:48:43.905575 ip-10-0-128-131 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 16:48:53.921140 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 16:48:53.921160 ip-10-0-128-131 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bd16ee5fe4654b809cbdb8c4f15c2e2d --
Jul 09 16:50:58.140357 ip-10-0-128-131 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 16:50:58.573153 ip-10-0-128-131 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:50:58.573153 ip-10-0-128-131 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 16:50:58.573153 ip-10-0-128-131 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:50:58.573153 ip-10-0-128-131 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 16:50:58.573153 ip-10-0-128-131 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:50:58.574171 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.574081    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 16:50:58.577537 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577519    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:58.577537 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577536    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577540    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577543    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577546    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577549    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577552    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577555    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577558    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577561    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577563    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577566    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577569    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577572    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577576    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577578    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577581    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577583    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577586    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577589    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:58.577609 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577592    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577594    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577597    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577600    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577602    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577605    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577607    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577609    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577612    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577619    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577622    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577625    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577628    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577631    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577633    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577636    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577638    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577641    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577645    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:58.578133 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577649    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577652    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577654    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577657    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577659    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577662    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577664    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577667    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577669    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577671    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577674    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577676    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577678    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577680    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577685    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577690    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577693    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577696    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577699    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:58.578687 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577702    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577705    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577708    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577711    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577714    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577716    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577719    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577721    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577724    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577726    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577729    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577731    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577734    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577736    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577739    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577741    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577744    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577747    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577750    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577753    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:58.579230 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577756    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577759    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577763    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577766    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577769    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577771    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577774    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577776    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577779    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577781    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577784    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577786    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577789    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577791    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577794    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577796    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577799    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577801    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577804    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577806    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:58.579782 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.577809    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578511    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578517    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578520    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578523    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578525    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578528    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578530    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578534    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578536    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578539    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578541    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578544    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578548    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578550    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578553    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578558    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578562    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578565    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:58.580342 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578568    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578571    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578574    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578577    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578580    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578583    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578586    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578588    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578591    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578593    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578596    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578599    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578601    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578604    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578606    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578609    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578611    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578614    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578616    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578619    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:58.580886 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578621    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578623    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578626    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578628    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578631    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578635    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578637    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578640    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578642    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578645    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578649    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578652    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578655    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578660    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578663    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578665    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578668    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578670    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578673    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578675    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:58.581460 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578678    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578681    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578683    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578686    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578688    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578691    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578695    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578697    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578700    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578703    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578705    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578708    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578710    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578713    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578715    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578717    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578720    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578723    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578725    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578728    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:58.582022 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578730    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578733    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578736    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578738    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578742    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578744    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578747    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578749    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578752    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578754    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578757    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578761    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578765    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578767    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578770    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578773    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578776    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578778    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578781    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578783    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:58.582597 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.578786    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579173    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579179    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579181    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579184    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579187    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579189    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579192    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579195    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579198    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579201    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579203    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579206    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579208    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579211    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579214    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579217    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579220    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579222    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:58.583186 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579225    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579227    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579230    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579232    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579235    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579237    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579240    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579245    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579248    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579251    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579254    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579256    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579259    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579262    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579264    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579267    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579269    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579271    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579274    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579276    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:58.583700 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579279    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579281    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579284    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579287    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579289    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579292    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579296    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579298    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579301    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579303    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579306    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579308    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579310    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579313    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579315    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579317    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579320    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579322    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579325    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579330    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:58.584357 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579333    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579335    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579337    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579340    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579342    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579345    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579348    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579351    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579355    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579357    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579360    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579363    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579365    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579367    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579370    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579372    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579374    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579377    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579379    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:58.584937 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579382    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579384    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579389    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579392    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579395    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579397    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579400    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579403    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579405    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579408    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579410    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579413    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579417    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579419    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579422    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579424    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579426    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579429    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579431    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:58.585478 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579434    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579437    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.579439    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579518    2567 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579534    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579543    2567 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579549    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579555    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579560    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579567    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579572    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579575    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579578    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579581    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579585    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579588    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579591    2567 flags.go:64] FLAG: --cgroup-root=""
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579594    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579597    2567 flags.go:64] FLAG: --client-ca-file=""
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579600    2567 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579603    2567 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579608    2567 flags.go:64] FLAG: --cluster-domain=""
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579610    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579613    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 16:50:58.586000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579616    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579621    2567 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579625    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579628    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579631    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579634    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579637    2567 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579640    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579643    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579646    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579649    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579656    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579659    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579662    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579665    2567 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579667    2567 flags.go:64] FLAG: --enable-server="true"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579672    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579676    2567 flags.go:64] FLAG: --event-burst="100"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579679    2567 flags.go:64] FLAG: --event-qps="50"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579682    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579685    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579688    2567 flags.go:64] FLAG: --eviction-hard=""
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579691    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579695    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579698    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 16:50:58.586757 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579702    2567 flags.go:64] FLAG: --eviction-soft=""
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579705    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579707    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579710    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579713    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579716    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579719    2567 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579721    2567 flags.go:64] FLAG: --feature-gates=""
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579726    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579729    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579732    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579735    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579738    2567 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579741    2567 flags.go:64] FLAG: --help="false"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579743    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579746    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579749    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579752    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579755    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579759    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579761    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579764    2567 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579767    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 16:50:58.587452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579772    2567 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579774    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579778    2567 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579780    2567 flags.go:64] FLAG: --kube-reserved=""
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579783    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579786    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579789    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579792    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579795    2567 flags.go:64] FLAG: --lock-file=""
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579798    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579801    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579805    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579810    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579813    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579815    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579818    2567 flags.go:64] FLAG: --logging-format="text"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579821    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579825    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579828    2567 flags.go:64] FLAG: --manifest-url=""
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579831    2567 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579835    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579838    2567 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579841    2567 flags.go:64] FLAG: --max-pods="110"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579844    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579847    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 16:50:58.588105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579850    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579853    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579856    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579859    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579861    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579868    2567 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579871    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579876    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579879    2567 flags.go:64] FLAG: --pod-cidr=""
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579882    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579887    2567 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579890    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579893    2567 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579896    2567 flags.go:64] FLAG: --port="10250"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579898    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579901    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02a9d80f9dcfb6671"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579904    2567 flags.go:64] FLAG: --qos-reserved=""
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579907    2567 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579910    2567 flags.go:64] FLAG: --register-node="true"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579913    2567 flags.go:64] FLAG: --register-with-taints=""
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579917    2567 flags.go:64] FLAG: --registry-burst="10"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579920    2567 flags.go:64] FLAG: --registry-qps="5"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579923    2567 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579925    2567 flags.go:64] FLAG: --reserved-memory=""
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579930    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 16:50:58.588778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579933    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579936    2567 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579939    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579941    2567 flags.go:64] FLAG: --runonce="false"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579944    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579947    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579950    2567 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579953    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579956    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579959    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579962    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579965    2567 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579968    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579971    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579974    2567 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579978    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579981    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579984    2567 flags.go:64] FLAG: --system-cgroups=""
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579987    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579992    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579995    2567 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.579997    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580002    2567 flags.go:64] FLAG: --tls-min-version=""
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580005    2567 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580007    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 16:50:58.589491 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580010    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580018    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580021    2567 flags.go:64] FLAG: --v="2"
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580024    2567 flags.go:64] FLAG: --version="false"
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580028    2567 flags.go:64] FLAG: --vmodule=""
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580032    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.580036    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580207    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580212    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580215    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580219    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580221    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580224    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580226    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580229    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580232    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580234    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580237    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580240    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580242    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580245    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:58.590233 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580247    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580252    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580266    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580270    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580273    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580275    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580278    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580281    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580285    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580289    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580292    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580294    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580297    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580300    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580302    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580304    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580307    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580312    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580315    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580318    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:58.590821 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580320    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580322    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580325    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580327    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580329    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580332    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580334    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580337    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580340    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580342    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580344    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580347    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580350    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580353    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580356    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580359    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580361    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580364    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580367    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580369    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:58.591437 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580372    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580374    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580377    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580379    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580382    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580385    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580388    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580390    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580393    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580396    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580399    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580401    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580404    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580407    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580411    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580413    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580416    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580419    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580422    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:58.591988 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580425    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580428    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580431    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580433    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580436    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580438    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580442    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580445    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580448    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580450    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580453    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580455    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580458    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580461    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580465    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580468    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580470    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580473    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580475    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:58.592547 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580478    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580481    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580484    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580487    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580489    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580492    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.580494    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.581143    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588024    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588040    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588057    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588078    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588142    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588147    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588151    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:58.593237 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588155    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588159    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588162    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588165    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588167    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588170    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588173    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588175    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588178    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588181    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588183    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588186    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588189    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588191    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588194    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588196    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588199    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588201    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588204    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:58.593674 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588206    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588208    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588211    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588214    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588217    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588219    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588222    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588226    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588229    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588231    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588234    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588236    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588239    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588241    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588244    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588247    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588249    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588251    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588254    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:58.594214 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588258    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588261    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588264    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588267    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588270    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588273    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588275    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588278    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588280    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588283    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588286    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588289    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588291    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588294    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588296    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588299    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588301    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588304    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588306    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588308    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:58.594740 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588312    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588316    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588318    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588321    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588323    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588326    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588330    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588333    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588335    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588338    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588340    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588343    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588345    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588348    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588350    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588353    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588355    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588358    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588360    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:58.595292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588363    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588365    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588368    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588370    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588372    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588375    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588377    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588379    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588382    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588384    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588387    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588389    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588391    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588394    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588397    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588400    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588403    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588405    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:58.595803 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588407    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588413    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588550    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588556    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588558    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588561    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588564    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588567    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588570    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588572    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588575    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588578    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588580    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588583    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588586    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:58.596313 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588588    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588591    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588593    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588596    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588598    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588601    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588603    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588606    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588608    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588611    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588613    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588615    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588620    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588624    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588627    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588635    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588637    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588640    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588644    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:58.596756 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588647    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588650    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588653    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588655    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588658    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588660    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588663    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588665    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588668    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588670    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588673    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588675    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588677    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588680    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588682    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588685    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588688    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588690    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588693    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588695    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:58.597300 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588697    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588700    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588702    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588705    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588707    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588710    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588713    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588715    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588718    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588721    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588723    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588726    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588728    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588731    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588733    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588735    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588738    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588740    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588742    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588745    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:58.597848 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588747    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588750    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588752    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588754    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588757    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588759    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588763    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588766    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588769    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588772    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588775    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588778    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588781    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588784    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588787    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588790    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588792    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588796    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588799    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:58.598474 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588802    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588804    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588807    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588810    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588812    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588814    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588817    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:50:58.588819    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588824    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.588933    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.590956    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.592340    2567 server.go:1019] "Starting client certificate rotation"
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.592446    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 16:50:58.598994 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.593108    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 16:50:58.599373 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.596350    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 16:50:58.599373 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.599304    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 16:50:58.612040 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.612016    2567 log.go:25] "Validated CRI v1 runtime API"
Jul 09 16:50:58.614216 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.614192    2567 log.go:25] "Validated CRI v1 image API"
Jul 09 16:50:58.615742 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.615727    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 16:50:58.618445 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.618424    2567 fs.go:135] Filesystem UUIDs: map[3a3ed829-f41d-4a35-80b6-74bb342a31e7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8fa249dc-0c6b-41da-b098-a68650754798:/dev/nvme0n1p4]
Jul 09 16:50:58.618503 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.618445    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 16:50:58.623764 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.623746    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 16:50:58.624445 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.624328    2567 manager.go:216] Machine: {Timestamp:2026-07-09 16:50:58.622488648 +0000 UTC m=+0.358391968 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098569 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246a04a5e82885f4ed2d761d0ed0b7 SystemUUID:ec246a04-a5e8-2885-f4ed-2d761d0ed0b7 BootID:bd16ee5f-e465-4b80-9cbd-b8c4f15c2e2d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cc:86:0f:e1:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cc:86:0f:e1:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:af:ba:ea:34:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 16:50:58.624520 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.624448    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 16:50:58.624554 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.624539    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 16:50:58.628503 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.628479    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 16:50:58.628660 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.628507    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 16:50:58.629384 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.629373    2567 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 16:50:58.629416 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.629387    2567 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 16:50:58.629416 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.629395    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 16:50:58.631138 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.631128    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 16:50:58.632279 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.632269    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 16:50:58.635149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635136    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 16:50:58.635277 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635267    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 16:50:58.635305 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635299    2567 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 16:50:58.635332 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635308    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 16:50:58.635332 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635319    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 16:50:58.635332 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635329    2567 kubelet.go:398] "Adding apiserver pod source"
Jul 09 16:50:58.635422 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635339    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 16:50:58.635777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635748    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 16:50:58.635828 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.635788    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 16:50:58.638103 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.638079    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 16:50:58.639462 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.639447    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 16:50:58.639547 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.639478    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 16:50:58.640894 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640880    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640899    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640906    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640912    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640921    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640926    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640934    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640939    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640947    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 16:50:58.640950 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640953    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 16:50:58.641261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640971    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 16:50:58.641261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.640989    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 16:50:58.642616 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.642601    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 16:50:58.642616 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.642613    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 16:50:58.644293 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.644269    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 16:50:58.644350 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.644316    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 16:50:58.644515 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.644487    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 16:50:58.646266 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.646253    2567 server.go:1268] "Started kubelet"
Jul 09 16:50:58.646426 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.646355    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 16:50:58.646542 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.646452    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 16:50:58.646604 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.646399    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 16:50:58.647143 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.647121    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zggb"
Jul 09 16:50:58.647376 ip-10-0-128-131 systemd[1]: Started Kubernetes Kubelet.
Jul 09 16:50:58.647747 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.647726    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 16:50:58.653513 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.653490    2567 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 16:50:58.653926 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.653899    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 16:50:58.654559 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.654537    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zggb"
Jul 09 16:50:58.657869 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.657852    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 16:50:58.657953 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.657880    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 16:50:58.658006 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.657966    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:58.658458 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.658424    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 16:50:58.658458 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.658434    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 16:50:58.658458 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.658440    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 16:50:58.658620 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.658552    2567 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 16:50:58.658620 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.658585    2567 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 16:50:58.659277 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659078    2567 factory.go:55] Registering systemd factory
Jul 09 16:50:58.659277 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659246    2567 factory.go:223] Registration of the systemd container factory successfully
Jul 09 16:50:58.659531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659479    2567 factory.go:153] Registering CRI-O factory
Jul 09 16:50:58.659531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659490    2567 factory.go:223] Registration of the crio container factory successfully
Jul 09 16:50:58.659621 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659590    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 16:50:58.659621 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659615    2567 factory.go:103] Registering Raw factory
Jul 09 16:50:58.659712 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.659627    2567 manager.go:1196] Started watching for new ooms in manager
Jul 09 16:50:58.660023 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.660008    2567 manager.go:349] Starting recovery of all containers
Jul 09 16:50:58.664391 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.664347    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:58.667598 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.667576    2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.670992 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.670975    2567 manager.go:354] Recovery completed
Jul 09 16:50:58.674952 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.674941    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.676914 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.676900    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.676966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.676928    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.676966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.676938    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.677504 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.677489    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 16:50:58.677504 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.677502    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 16:50:58.677650 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.677522    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 16:50:58.677650 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.677495    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.678266 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.678252    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.678322 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.678279    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.678322 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.678289    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.678898 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.678887    2567 policy_none.go:49] "None policy: Start"
Jul 09 16:50:58.678931 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.678903    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 16:50:58.678931 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.678914    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 16:50:58.679923 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.679912    2567 policy_none.go:47] "Start"
Jul 09 16:50:58.719738 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.719720    2567 manager.go:337] "Starting Device Plugin manager"
Jul 09 16:50:58.719876 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.719755    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 16:50:58.719876 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.719765    2567 server.go:87] "Starting device plugin registration server"
Jul 09 16:50:58.720040 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.720024    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 16:50:58.720154 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.720038    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 16:50:58.720221 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.720206    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 16:50:58.720291 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.720275    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 16:50:58.720291 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.720286    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 16:50:58.720810 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.720787    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 16:50:58.720893 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.720836    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:58.820718 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.820664    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.822305 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.822008    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.822305 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.822039    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.822305 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.822050    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.822305 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.822097    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.825924 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.825857    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 16:50:58.827156 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.827137    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 16:50:58.827249 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.827180    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 16:50:58.827249 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.827203    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 16:50:58.827249 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.827237    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 16:50:58.829875 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.829848    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:58.832125 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.832109    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.832208 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.832133    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-131.ec2.internal\": node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:58.879713 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.879678    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:58.927786 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.927747    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"]
Jul 09 16:50:58.927899 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.927844    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.928910 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.928891    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.929041 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.928926    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.929041 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.928941    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.930252 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.930236    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.930392 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.930378    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.930432 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.930411    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.930979 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.930958    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.930979 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.930981    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.931093 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.930994    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.931093 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.931031    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.931093 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.931055    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.931093 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.931089    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.932197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.932183    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.932257 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.932209    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:58.932822 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.932805    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:58.932914 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.932837    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:58.932914 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.932868    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:58.960432 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.960400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bce01f930a010af7670ca85e9b0780aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"bce01f930a010af7670ca85e9b0780aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.960432 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.960430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c2bdb575537e3e4dfbbc28296b46e7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"2c2bdb575537e3e4dfbbc28296b46e7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.960605 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:58.960450    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c2bdb575537e3e4dfbbc28296b46e7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"2c2bdb575537e3e4dfbbc28296b46e7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.965977 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.965962    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.970696 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.970680    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal"
Jul 09 16:50:58.979749 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:58.979730    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:59.060997 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.060936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bce01f930a010af7670ca85e9b0780aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"bce01f930a010af7670ca85e9b0780aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.061141 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.061010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c2bdb575537e3e4dfbbc28296b46e7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"2c2bdb575537e3e4dfbbc28296b46e7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.061141 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.061042    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c2bdb575537e3e4dfbbc28296b46e7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"2c2bdb575537e3e4dfbbc28296b46e7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.061141 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.060961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bce01f930a010af7670ca85e9b0780aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"bce01f930a010af7670ca85e9b0780aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.061141 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.061096    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c2bdb575537e3e4dfbbc28296b46e7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"2c2bdb575537e3e4dfbbc28296b46e7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.061141 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.061131    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c2bdb575537e3e4dfbbc28296b46e7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"2c2bdb575537e3e4dfbbc28296b46e7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.080099 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:59.080034    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:59.180869 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:59.180836    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:50:59.269367 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.269327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.274392 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:50:59.274372    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Jul 09 16:50:59.281393 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:50:59.281371    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.420715 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:00.420625    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.521164 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:00.521130    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.621753 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:00.621722    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.630916 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.630888    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 16:51:00.631096 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.631075    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 16:51:00.631138 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.631075    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 16:51:00.698287 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.698192    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 16:45:59 +0000 UTC" deadline="2028-03-15 11:48:37.224035336 +0000 UTC"
Jul 09 16:51:00.698287 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.698218    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14754h57m36.525819849s"
Jul 09 16:51:00.721958 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:00.721932    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.822583 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:00.822543    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.923049 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:00.922994    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Jul 09 16:51:00.988135 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.988066    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:51:00.997185 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:00.997156    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Jul 09 16:51:01.007776 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.007751    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 16:51:01.009673 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.009652    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Jul 09 16:51:01.015846 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.015829    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 16:51:01.127545 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.127521    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:51:01.213184 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:01.213152    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce01f930a010af7670ca85e9b0780aa.slice/crio-ed9199b0d9101f1b88189aa77ff426b3dce9d9700f80bb5f9bce3bf1433a487a WatchSource:0}: Error finding container ed9199b0d9101f1b88189aa77ff426b3dce9d9700f80bb5f9bce3bf1433a487a: Status 404 returned error can't find the container with id ed9199b0d9101f1b88189aa77ff426b3dce9d9700f80bb5f9bce3bf1433a487a
Jul 09 16:51:01.219201 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.219180    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:51:01.222489 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:01.222454    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2bdb575537e3e4dfbbc28296b46e7d.slice/crio-02461bf48e27852002fa7b819dacc09b68ffc3d84f7bcd59f8a89c109b91b21d WatchSource:0}: Error finding container 02461bf48e27852002fa7b819dacc09b68ffc3d84f7bcd59f8a89c109b91b21d: Status 404 returned error can't find the container with id 02461bf48e27852002fa7b819dacc09b68ffc3d84f7bcd59f8a89c109b91b21d
Jul 09 16:51:01.667641 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.667605    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:51:01.675767 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.675746    2567 apiserver.go:52] "Watching apiserver"
Jul 09 16:51:01.683391 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.683369    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 16:51:01.684331 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.684309    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vp2jf","openshift-dns/node-resolver-dqmmt","openshift-image-registry/node-ca-2fhkn","openshift-multus/multus-9fdgf","openshift-multus/multus-additional-cni-plugins-brnqb","openshift-network-diagnostics/network-check-target-bh5ql","openshift-network-operator/iptables-alerter-xnpm5","openshift-ovn-kubernetes/ovnkube-node-5ffwd","kube-system/global-pull-secret-syncer-4fx9l","kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl","openshift-cluster-node-tuning-operator/tuned-5jqxb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal","openshift-multus/network-metrics-daemon-884hx"]
Jul 09 16:51:01.685970 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.685955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.687382 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.687357    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.688419 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.688399    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h5lwr\""
Jul 09 16:51:01.688419 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.688405    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.688580 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.688446    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 16:51:01.688580 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.688482    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.688753 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.688741    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.690008 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.689988    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.690772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.690387    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 16:51:01.690772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.690540    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.692372 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.691326    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2tjj5\""
Jul 09 16:51:01.692372 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.691369    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.692372 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.691409    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sstx6\""
Jul 09 16:51:01.692372 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.692207    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 16:51:01.692372 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.692214    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.692659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.692466    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.693710 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.693681    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.694057 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.694027    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.694150 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.694113    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 16:51:01.694339 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.694322    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 16:51:01.694406 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.694326    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pqpkc\""
Jul 09 16:51:01.694688 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.694672    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.694967 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.694937    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:01.695047 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.695002    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:01.696036 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.696017    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w7h9m\""
Jul 09 16:51:01.696130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.696021    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 16:51:01.696130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.696051    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 16:51:01.696261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.696244    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.696346 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.696326    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:01.697468 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.697454    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 16:51:01.697584 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.697553    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:01.697646 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.697627    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:01.698904 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.698886    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.700216 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.700196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.701243 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.701225    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.701350 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.701230    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6zt8\""
Jul 09 16:51:01.701664 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.701647    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.702383 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.702274    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 16:51:01.702763 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.702746    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.702846 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.702819    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 16:51:01.702846 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.702827    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.702938 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.702910    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fv94\""
Jul 09 16:51:01.703043 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.702933    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 16:51:01.703094 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.703049    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 16:51:01.703094 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.703063    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.703552 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.703534    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.704072 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.704055    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-btp7k\""
Jul 09 16:51:01.704149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.704116    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 16:51:01.705353 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.705333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 16:51:01.706101 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.706083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 16:51:01.706249 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.706227    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:51:01.706452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.706439    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tvbt7\""
Jul 09 16:51:01.707924 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.707908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-socket-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.708003 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.707938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:01.708003 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.707966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhht\" (UniqueName: \"kubernetes.io/projected/f4d2921f-10ef-49f7-a552-6e511cb4435f-kube-api-access-plhht\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.708003 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.707990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-cni-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-netns\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-conf-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708068    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.708149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.708149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-system-cni-dir\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.708149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708141    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cnibin\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-tmp-dir\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708191    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-host-slash\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lcz\" (UniqueName: \"kubernetes.io/projected/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-kube-api-access-67lcz\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708284    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbqz\" (UniqueName: \"kubernetes.io/projected/9260de68-1f8b-475a-a8af-c8cb571c38f7-kube-api-access-jvbqz\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4d2921f-10ef-49f7-a552-6e511cb4435f-serviceca\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708331    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-daemon-config\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708359    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gp94\" (UniqueName: \"kubernetes.io/projected/6bc945b8-7b27-42e2-b286-4020fdbd916f-kube-api-access-6gp94\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708417 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708386    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-device-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4d2921f-10ef-49f7-a552-6e511cb4435f-host\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708486    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-cnibin\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-k8s-cni-cncf-io\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-cni-bin\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-kubelet\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8czc\" (UniqueName: \"kubernetes.io/projected/60d6296c-10c8-498f-b402-cfbf51a4f1b3-kube-api-access-s8czc\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708612    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77fbfceb-30f2-4e00-9a5f-c083f062c845-dbus\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-os-release\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-cni-multus\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708685    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-multus-certs\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfcs\" (UniqueName: \"kubernetes.io/projected/2cff268e-e6cc-4a37-bac8-b79cd442a131-kube-api-access-pdfcs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.708788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708783    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-hosts-file\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708848    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pszv\" (UniqueName: \"kubernetes.io/projected/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-kube-api-access-4pszv\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-iptables-alerter-script\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708950    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bc945b8-7b27-42e2-b286-4020fdbd916f-cni-binary-copy\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.708983    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-etc-kubernetes\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709010    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-os-release\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709096    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-socket-dir-parent\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709125    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-registration-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709147    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-sys-fs\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709223    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77fbfceb-30f2-4e00-9a5f-c083f062c845-kubelet-config\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-system-cni-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.709328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.709307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-hostroot\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.710314 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.710280    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 16:51:01.716810 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.716778    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:51:01.730897 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.730872    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rrdkr"
Jul 09 16:51:01.737902 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.737877    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rrdkr"
Jul 09 16:51:01.798642 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.798613    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 16:51:01.810014 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.809978    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810052    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-system-cni-dir\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cnibin\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810106    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-node-log\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-cni-bin\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810126    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810125    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.810194 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-run\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810197    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cnibin\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gp94\" (UniqueName: \"kubernetes.io/projected/6bc945b8-7b27-42e2-b286-4020fdbd916f-kube-api-access-6gp94\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-ovn\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810190    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-system-cni-dir\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-cni-netd\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-slash\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-device-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4d2921f-10ef-49f7-a552-6e511cb4435f-host\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810394    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-k8s-cni-cncf-io\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-device-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-k8s-cni-cncf-io\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810485    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4d2921f-10ef-49f7-a552-6e511cb4435f-host\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810517    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b895e80b-97a9-499b-bc6f-2d10ede4be8c-agent-certs\") pod \"konnectivity-agent-vp2jf\" (UID: \"b895e80b-97a9-499b-bc6f-2d10ede4be8c\") " pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-host\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.810572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810574    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8czc\" (UniqueName: \"kubernetes.io/projected/60d6296c-10c8-498f-b402-cfbf51a4f1b3-kube-api-access-s8czc\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77fbfceb-30f2-4e00-9a5f-c083f062c845-dbus\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-os-release\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810869    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-os-release\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810872    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysconfig\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810914    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77fbfceb-30f2-4e00-9a5f-c083f062c845-dbus\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810946    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-hosts-file\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810977    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-hosts-file\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.810994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-iptables-alerter-script\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bc945b8-7b27-42e2-b286-4020fdbd916f-cni-binary-copy\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811057    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-var-lib-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3afc9352-6013-476b-97ed-7e2834837109-ovn-node-metrics-cert\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysctl-d\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-tuned\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-os-release\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-socket-dir-parent\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.811381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-systemd-units\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811278    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-systemd\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-var-lib-kubelet\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zn6s\" (UniqueName: \"kubernetes.io/projected/2cf2914d-6687-41b0-9c38-40d40d38d888-kube-api-access-7zn6s\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-registration-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77fbfceb-30f2-4e00-9a5f-c083f062c845-kubelet-config\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811451    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-os-release\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-socket-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811511    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-cni-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-netns\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-tmp-dir\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811596    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-host-slash\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lcz\" (UniqueName: \"kubernetes.io/projected/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-kube-api-access-67lcz\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811665    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-kubelet\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.812210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-run-netns\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811722    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbqz\" (UniqueName: \"kubernetes.io/projected/9260de68-1f8b-475a-a8af-c8cb571c38f7-kube-api-access-jvbqz\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4d2921f-10ef-49f7-a552-6e511cb4435f-serviceca\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-daemon-config\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-env-overrides\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811887    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-cnibin\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-cni-bin\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-kubelet\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-etc-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.811990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b895e80b-97a9-499b-bc6f-2d10ede4be8c-konnectivity-ca\") pod \"konnectivity-agent-vp2jf\" (UID: \"b895e80b-97a9-499b-bc6f-2d10ede4be8c\") " pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-modprobe-d\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cf2914d-6687-41b0-9c38-40d40d38d888-tmp\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-cni-multus\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812075    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bc945b8-7b27-42e2-b286-4020fdbd916f-cni-binary-copy\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-multus-certs\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813061 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812116    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-host-slash\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfcs\" (UniqueName: \"kubernetes.io/projected/2cff268e-e6cc-4a37-bac8-b79cd442a131-kube-api-access-pdfcs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-ovnkube-script-lib\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.812198    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.812289    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:02.312249638 +0000 UTC m=+3.009421961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812393    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77fbfceb-30f2-4e00-9a5f-c083f062c845-kubelet-config\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812472    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-socket-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812609    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-socket-dir-parent\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812696    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-registration-dir\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-iptables-alerter-script\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4d2921f-10ef-49f7-a552-6e511cb4435f-serviceca\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813072    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-cni-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813104    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-cni-multus\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-netns\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813167    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-run-multus-certs\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.813866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.812203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pszv\" (UniqueName: \"kubernetes.io/projected/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-kube-api-access-4pszv\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-etc-kubernetes\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-tmp-dir\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-systemd\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5w4\" (UniqueName: \"kubernetes.io/projected/3afc9352-6013-476b-97ed-7e2834837109-kube-api-access-pp5w4\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-ovnkube-config\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysctl-conf\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-sys-fs\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813399    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-cnibin\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813458    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9260de68-1f8b-475a-a8af-c8cb571c38f7-sys-fs\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-system-cni-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813492    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-hostroot\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813511    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-kubernetes\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-cni-bin\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.814655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813579    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-etc-kubernetes\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60d6296c-10c8-498f-b402-cfbf51a4f1b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.813684    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-sys\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813702    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-daemon-config\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813742    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-system-cni-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-lib-modules\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813782    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-host-var-lib-kubelet\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.813828    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:02.313811473 +0000 UTC m=+3.010983792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813852    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhht\" (UniqueName: \"kubernetes.io/projected/f4d2921f-10ef-49f7-a552-6e511cb4435f-kube-api-access-plhht\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-conf-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.813999    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-hostroot\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.814005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-log-socket\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.814046    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bc945b8-7b27-42e2-b286-4020fdbd916f-multus-conf-dir\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.815348 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.814251    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60d6296c-10c8-498f-b402-cfbf51a4f1b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.816977 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.816944    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 16:51:01.820889 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.820868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8czc\" (UniqueName: \"kubernetes.io/projected/60d6296c-10c8-498f-b402-cfbf51a4f1b3-kube-api-access-s8czc\") pod \"multus-additional-cni-plugins-brnqb\" (UID: \"60d6296c-10c8-498f-b402-cfbf51a4f1b3\") " pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:01.821008 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.820886    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gp94\" (UniqueName: \"kubernetes.io/projected/6bc945b8-7b27-42e2-b286-4020fdbd916f-kube-api-access-6gp94\") pod \"multus-9fdgf\" (UID: \"6bc945b8-7b27-42e2-b286-4020fdbd916f\") " pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:01.821420 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.821377    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:01.821420 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.821407    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:01.821420 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.821420    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:01.821619 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:01.821479    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:02.321466094 +0000 UTC m=+3.018638398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:01.821829 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.821751    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhht\" (UniqueName: \"kubernetes.io/projected/f4d2921f-10ef-49f7-a552-6e511cb4435f-kube-api-access-plhht\") pod \"node-ca-2fhkn\" (UID: \"f4d2921f-10ef-49f7-a552-6e511cb4435f\") " pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:01.822401 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.822033    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pszv\" (UniqueName: \"kubernetes.io/projected/9c8b2164-2ab2-4a8f-9549-1ebf5a50800e-kube-api-access-4pszv\") pod \"node-resolver-dqmmt\" (UID: \"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e\") " pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:01.823992 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.823970    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lcz\" (UniqueName: \"kubernetes.io/projected/8b308928-5e0e-4beb-aa2d-1f4fe94348e5-kube-api-access-67lcz\") pod \"iptables-alerter-xnpm5\" (UID: \"8b308928-5e0e-4beb-aa2d-1f4fe94348e5\") " pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:01.824412 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.824379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfcs\" (UniqueName: \"kubernetes.io/projected/2cff268e-e6cc-4a37-bac8-b79cd442a131-kube-api-access-pdfcs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:01.824585 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.824568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbqz\" (UniqueName: \"kubernetes.io/projected/9260de68-1f8b-475a-a8af-c8cb571c38f7-kube-api-access-jvbqz\") pod \"aws-ebs-csi-driver-node-cbfzl\" (UID: \"9260de68-1f8b-475a-a8af-c8cb571c38f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:01.870265 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.870192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"2c2bdb575537e3e4dfbbc28296b46e7d","Type":"ContainerStarted","Data":"02461bf48e27852002fa7b819dacc09b68ffc3d84f7bcd59f8a89c109b91b21d"}
Jul 09 16:51:01.871116 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.871087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" event={"ID":"bce01f930a010af7670ca85e9b0780aa","Type":"ContainerStarted","Data":"ed9199b0d9101f1b88189aa77ff426b3dce9d9700f80bb5f9bce3bf1433a487a"}
Jul 09 16:51:01.915456 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-kubernetes\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-sys\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-lib-modules\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-log-socket\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-node-log\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-cni-bin\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-run\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-ovn\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.915638 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-cni-netd\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-slash\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b895e80b-97a9-499b-bc6f-2d10ede4be8c-agent-certs\") pod \"konnectivity-agent-vp2jf\" (UID: \"b895e80b-97a9-499b-bc6f-2d10ede4be8c\") " pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-host\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysconfig\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-var-lib-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915819    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3afc9352-6013-476b-97ed-7e2834837109-ovn-node-metrics-cert\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysctl-d\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915864    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-tuned\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-systemd-units\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-systemd\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915959    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-var-lib-kubelet\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.915982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zn6s\" (UniqueName: \"kubernetes.io/projected/2cf2914d-6687-41b0-9c38-40d40d38d888-kube-api-access-7zn6s\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916030    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-kubelet\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916062 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916053    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-run-netns\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916080    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-env-overrides\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916108    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-etc-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b895e80b-97a9-499b-bc6f-2d10ede4be8c-konnectivity-ca\") pod \"konnectivity-agent-vp2jf\" (UID: \"b895e80b-97a9-499b-bc6f-2d10ede4be8c\") " pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-modprobe-d\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916177    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cf2914d-6687-41b0-9c38-40d40d38d888-tmp\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-ovnkube-script-lib\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-systemd\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5w4\" (UniqueName: \"kubernetes.io/projected/3afc9352-6013-476b-97ed-7e2834837109-kube-api-access-pp5w4\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-ovnkube-config\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysctl-conf\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916551    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysctl-conf\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916616    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-kubernetes\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916662    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-sys\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916760    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-lib-modules\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.916831 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916811    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-log-socket\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-node-log\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916885    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-cni-bin\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916926    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-run\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916957    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-ovn\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.916992    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-cni-netd\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-slash\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.917537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-kubelet\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918009 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917540    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-etc-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918009 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917543    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-run-netns\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918009 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-host\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.918009 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysconfig\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.918009 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.917680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-var-lib-openvswitch\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918245 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-env-overrides\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918245 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918150    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918245 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-systemd-units\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918389 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918197    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-var-lib-kubelet\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.918389 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918363    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-systemd\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.918485 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918439    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-run-systemd\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918539 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918521    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-sysctl-d\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.918592 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918571    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3afc9352-6013-476b-97ed-7e2834837109-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918706 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-ovnkube-script-lib\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.918811 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.918755    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-modprobe-d\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.919122 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.919085    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b895e80b-97a9-499b-bc6f-2d10ede4be8c-konnectivity-ca\") pod \"konnectivity-agent-vp2jf\" (UID: \"b895e80b-97a9-499b-bc6f-2d10ede4be8c\") " pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.919200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.919116    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3afc9352-6013-476b-97ed-7e2834837109-ovnkube-config\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.921456 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.921428    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b895e80b-97a9-499b-bc6f-2d10ede4be8c-agent-certs\") pod \"konnectivity-agent-vp2jf\" (UID: \"b895e80b-97a9-499b-bc6f-2d10ede4be8c\") " pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:01.921688 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.921662    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cf2914d-6687-41b0-9c38-40d40d38d888-tmp\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.922331 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.922308    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3afc9352-6013-476b-97ed-7e2834837109-ovn-node-metrics-cert\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:01.922768 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.922745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2cf2914d-6687-41b0-9c38-40d40d38d888-etc-tuned\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.928099 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.928072    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zn6s\" (UniqueName: \"kubernetes.io/projected/2cf2914d-6687-41b0-9c38-40d40d38d888-kube-api-access-7zn6s\") pod \"tuned-5jqxb\" (UID: \"2cf2914d-6687-41b0-9c38-40d40d38d888\") " pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:01.930252 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:01.930233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5w4\" (UniqueName: \"kubernetes.io/projected/3afc9352-6013-476b-97ed-7e2834837109-kube-api-access-pp5w4\") pod \"ovnkube-node-5ffwd\" (UID: \"3afc9352-6013-476b-97ed-7e2834837109\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:02.000332 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.000292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xnpm5"
Jul 09 16:51:02.005701 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.005676    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl"
Jul 09 16:51:02.012049 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.012011    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b308928_5e0e_4beb_aa2d_1f4fe94348e5.slice/crio-6f5970edf396228975b6a19dea6eba2c1929661829a6ea6b7ad32a7a27fdd8ad WatchSource:0}: Error finding container 6f5970edf396228975b6a19dea6eba2c1929661829a6ea6b7ad32a7a27fdd8ad: Status 404 returned error can't find the container with id 6f5970edf396228975b6a19dea6eba2c1929661829a6ea6b7ad32a7a27fdd8ad
Jul 09 16:51:02.012788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.012604    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2fhkn"
Jul 09 16:51:02.016938 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.016918    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fdgf"
Jul 09 16:51:02.017292 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.017267    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9260de68_1f8b_475a_a8af_c8cb571c38f7.slice/crio-084e8aa18908fcce710f4ebd5c15f64b8b9295dfe82b4359133266fc37ceac0e WatchSource:0}: Error finding container 084e8aa18908fcce710f4ebd5c15f64b8b9295dfe82b4359133266fc37ceac0e: Status 404 returned error can't find the container with id 084e8aa18908fcce710f4ebd5c15f64b8b9295dfe82b4359133266fc37ceac0e
Jul 09 16:51:02.022553 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.022532    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brnqb"
Jul 09 16:51:02.023189 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.023165    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d2921f_10ef_49f7_a552_6e511cb4435f.slice/crio-d934dfd1908d8ed5650759e750b6e7249812b06bf9e4fef6d61a1ebedca0efcf WatchSource:0}: Error finding container d934dfd1908d8ed5650759e750b6e7249812b06bf9e4fef6d61a1ebedca0efcf: Status 404 returned error can't find the container with id d934dfd1908d8ed5650759e750b6e7249812b06bf9e4fef6d61a1ebedca0efcf
Jul 09 16:51:02.026649 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.026628    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc945b8_7b27_42e2_b286_4020fdbd916f.slice/crio-4ea84400f3b1d461e73047e7969bafcf61e32b6c8798df9dd23ec18781906d40 WatchSource:0}: Error finding container 4ea84400f3b1d461e73047e7969bafcf61e32b6c8798df9dd23ec18781906d40: Status 404 returned error can't find the container with id 4ea84400f3b1d461e73047e7969bafcf61e32b6c8798df9dd23ec18781906d40
Jul 09 16:51:02.028130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.027990    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqmmt"
Jul 09 16:51:02.032977 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.032951    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d6296c_10c8_498f_b402_cfbf51a4f1b3.slice/crio-e35aaf468d9f94400a9a26501b88508933a1ea36652c9b83edfcf03042f10f2a WatchSource:0}: Error finding container e35aaf468d9f94400a9a26501b88508933a1ea36652c9b83edfcf03042f10f2a: Status 404 returned error can't find the container with id e35aaf468d9f94400a9a26501b88508933a1ea36652c9b83edfcf03042f10f2a
Jul 09 16:51:02.036882 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.036858    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:02.037145 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.037121    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8b2164_2ab2_4a8f_9549_1ebf5a50800e.slice/crio-eafff4ab30c3798e2f8e5a787cd487b203b071042c0229a1bb033566be1170b4 WatchSource:0}: Error finding container eafff4ab30c3798e2f8e5a787cd487b203b071042c0229a1bb033566be1170b4: Status 404 returned error can't find the container with id eafff4ab30c3798e2f8e5a787cd487b203b071042c0229a1bb033566be1170b4
Jul 09 16:51:02.044050 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.044031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:02.045151 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.045131    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afc9352_6013_476b_97ed_7e2834837109.slice/crio-cb9e569a9fa080561a1643695ea245c9782456c209b88546978d83308e86b0fc WatchSource:0}: Error finding container cb9e569a9fa080561a1643695ea245c9782456c209b88546978d83308e86b0fc: Status 404 returned error can't find the container with id cb9e569a9fa080561a1643695ea245c9782456c209b88546978d83308e86b0fc
Jul 09 16:51:02.050088 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.050060    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5jqxb"
Jul 09 16:51:02.051438 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.051420    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb895e80b_97a9_499b_bc6f_2d10ede4be8c.slice/crio-0d47c79ffd3913d0c58ac147feaec9f0a4548aef843260e155962c6a53830011 WatchSource:0}: Error finding container 0d47c79ffd3913d0c58ac147feaec9f0a4548aef843260e155962c6a53830011: Status 404 returned error can't find the container with id 0d47c79ffd3913d0c58ac147feaec9f0a4548aef843260e155962c6a53830011
Jul 09 16:51:02.058844 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:02.058729    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf2914d_6687_41b0_9c38_40d40d38d888.slice/crio-29563fd7c23f89ffb386f6a1ed25383edc913d8d47a047c4988a66d33572e6a2 WatchSource:0}: Error finding container 29563fd7c23f89ffb386f6a1ed25383edc913d8d47a047c4988a66d33572e6a2: Status 404 returned error can't find the container with id 29563fd7c23f89ffb386f6a1ed25383edc913d8d47a047c4988a66d33572e6a2
Jul 09 16:51:02.318973 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.318766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:02.318973 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.318990    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:02.319195 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.318917    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:02.319195 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.319097    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:03.319080771 +0000 UTC m=+4.016253103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:02.319195 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.319131    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:02.319195 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.319193    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:03.319177662 +0000 UTC m=+4.016349975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:02.420409 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.420367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:02.420586 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.420566    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:02.420637 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.420592    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:02.420637 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.420607    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:02.420726 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.420673    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:03.420651186 +0000 UTC m=+4.117823497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:02.739916 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.739869    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:46:01 +0000 UTC" deadline="2028-03-02 10:36:04.783928225 +0000 UTC"
Jul 09 16:51:02.739916 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.739913    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14441h45m2.044020322s"
Jul 09 16:51:02.866655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.866564    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:02.866839 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:02.866699    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:02.874514 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.874480    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqmmt" event={"ID":"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e","Type":"ContainerStarted","Data":"eafff4ab30c3798e2f8e5a787cd487b203b071042c0229a1bb033566be1170b4"}
Jul 09 16:51:02.877050 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.877021    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerStarted","Data":"e35aaf468d9f94400a9a26501b88508933a1ea36652c9b83edfcf03042f10f2a"}
Jul 09 16:51:02.879375 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.879343    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2fhkn" event={"ID":"f4d2921f-10ef-49f7-a552-6e511cb4435f","Type":"ContainerStarted","Data":"d934dfd1908d8ed5650759e750b6e7249812b06bf9e4fef6d61a1ebedca0efcf"}
Jul 09 16:51:02.882103 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.882074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vp2jf" event={"ID":"b895e80b-97a9-499b-bc6f-2d10ede4be8c","Type":"ContainerStarted","Data":"0d47c79ffd3913d0c58ac147feaec9f0a4548aef843260e155962c6a53830011"}
Jul 09 16:51:02.884878 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.884852    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"cb9e569a9fa080561a1643695ea245c9782456c209b88546978d83308e86b0fc"}
Jul 09 16:51:02.887070 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.887044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fdgf" event={"ID":"6bc945b8-7b27-42e2-b286-4020fdbd916f","Type":"ContainerStarted","Data":"4ea84400f3b1d461e73047e7969bafcf61e32b6c8798df9dd23ec18781906d40"}
Jul 09 16:51:02.890383 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.890357    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl" event={"ID":"9260de68-1f8b-475a-a8af-c8cb571c38f7","Type":"ContainerStarted","Data":"084e8aa18908fcce710f4ebd5c15f64b8b9295dfe82b4359133266fc37ceac0e"}
Jul 09 16:51:02.892463 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.892437    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xnpm5" event={"ID":"8b308928-5e0e-4beb-aa2d-1f4fe94348e5","Type":"ContainerStarted","Data":"6f5970edf396228975b6a19dea6eba2c1929661829a6ea6b7ad32a7a27fdd8ad"}
Jul 09 16:51:02.894315 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:02.894290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jqxb" event={"ID":"2cf2914d-6687-41b0-9c38-40d40d38d888","Type":"ContainerStarted","Data":"29563fd7c23f89ffb386f6a1ed25383edc913d8d47a047c4988a66d33572e6a2"}
Jul 09 16:51:03.329145 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.329058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:03.329145 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.329112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:03.329461 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.329345    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:03.329538 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.329521    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:05.329481953 +0000 UTC m=+6.026654258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:03.329969 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.329946    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:03.330060 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.330007    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:05.329990408 +0000 UTC m=+6.027162713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:03.430717 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.430341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:03.430717 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.430543    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:03.430717 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.430574    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:03.430717 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.430588    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:03.430717 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.430652    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:05.430630398 +0000 UTC m=+6.127802727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:03.740813 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.740729    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:46:01 +0000 UTC" deadline="2027-12-12 12:56:47.882625602 +0000 UTC"
Jul 09 16:51:03.740813 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.740767    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12500h5m44.141863303s"
Jul 09 16:51:03.866753 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.866724    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:03.866954 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.866877    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:03.867274 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.867251    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:03.867362 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:03.867340    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:03.910976 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:03.910108    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" event={"ID":"bce01f930a010af7670ca85e9b0780aa","Type":"ContainerStarted","Data":"16389ebb17ad57137c9dd4ce91b4a861e7c484cf7e6a3fed133dd220e4bc7479"}
Jul 09 16:51:04.866465 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:04.866375    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:04.866954 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:04.866523    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:04.919748 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:04.919711    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"2c2bdb575537e3e4dfbbc28296b46e7d","Type":"ContainerStarted","Data":"5cf8440eb8c9b5a3b3404328909206bfe8bea966ce11049707cd07cee940f7b6"}
Jul 09 16:51:04.934837 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:04.934760    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" podStartSLOduration=3.934739727 podStartE2EDuration="3.934739727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:03.929477635 +0000 UTC m=+4.626649972" watchObservedRunningTime="2026-07-09 16:51:04.934739727 +0000 UTC m=+5.631912055"
Jul 09 16:51:05.345211 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:05.345129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:05.345211 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:05.345180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:05.345434 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.345301    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:05.345434 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.345357    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:09.345338719 +0000 UTC m=+10.042511026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:05.345752 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.345732    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:05.345826 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.345808    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:09.345773624 +0000 UTC m=+10.042945931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:05.446220 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:05.445501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:05.446220 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.445711    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:05.446220 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.445733    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:05.446220 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.445746    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:05.446220 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.445824    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:09.445781429 +0000 UTC m=+10.142953738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:05.868421 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:05.868387    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:05.868865 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:05.868441    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:05.868865 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.868540    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:05.868865 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:05.868662    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:06.867054 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:06.867017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:06.867254 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:06.867156    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:07.866669 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:07.866581    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:07.867103 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:07.866737    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:07.867103 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:07.866791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:07.867103 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:07.866912    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:08.866508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:08.866471    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:08.866689 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:08.866587    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:09.377326 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.377239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:09.377326 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.377283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:09.377571 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.377397    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:09.377571 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.377445    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:17.377431827 +0000 UTC m=+18.074604133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:09.377773 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.377755    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:09.377881 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.377817    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:17.377785876 +0000 UTC m=+18.074958182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:09.477858 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.477820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:09.478077 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.477982    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:09.478077 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.478001    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:09.478077 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.478014    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:09.478077 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.478080    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:17.478061822 +0000 UTC m=+18.175234147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:09.869040 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.868925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:09.869552 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.869062    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:09.869552 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.869069    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:09.869552 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:09.869154    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:09.940493 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.940458    2567 generic.go:356] "Generic (PLEG): container finished" podID="2c2bdb575537e3e4dfbbc28296b46e7d" containerID="5cf8440eb8c9b5a3b3404328909206bfe8bea966ce11049707cd07cee940f7b6" exitCode=0
Jul 09 16:51:09.940679 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:09.940514    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"2c2bdb575537e3e4dfbbc28296b46e7d","Type":"ContainerDied","Data":"5cf8440eb8c9b5a3b3404328909206bfe8bea966ce11049707cd07cee940f7b6"}
Jul 09 16:51:10.866413 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:10.866377    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:10.866603 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:10.866500    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:11.867160 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:11.867124    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:11.867603 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:11.867259    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:11.867603 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:11.867320    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:11.867603 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:11.867454    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:12.866501 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:12.866465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:12.866664 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:12.866585    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:13.867938 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.867903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:13.868615 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:13.868025    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:13.869260 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.869236    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:13.869372 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:13.869350    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:13.952924 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.952885    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"2c2bdb575537e3e4dfbbc28296b46e7d","Type":"ContainerStarted","Data":"74ab78108c4c25c36afcb073368dc25267265529b6f46a7479149d8362fecca7"}
Jul 09 16:51:13.956038 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.955569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vp2jf" event={"ID":"b895e80b-97a9-499b-bc6f-2d10ede4be8c","Type":"ContainerStarted","Data":"c662d41e3c55cd08a2778caf76a2a615e993aba5314eb531ddbb1f9f67dc4eb7"}
Jul 09 16:51:13.957814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.957763    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl" event={"ID":"9260de68-1f8b-475a-a8af-c8cb571c38f7","Type":"ContainerStarted","Data":"268b9cd5d0ab027b0db7e33d0bd8d4a17fbb51ee1efc88bbbeeb60edfcaafe22"}
Jul 09 16:51:13.960064 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.959958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5jqxb" event={"ID":"2cf2914d-6687-41b0-9c38-40d40d38d888","Type":"ContainerStarted","Data":"ac71f0a61f86d81c5f06e3a3e029d0d7b421e3a9721ff7fd468b8af5bbecf612"}
Jul 09 16:51:13.962010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.961983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqmmt" event={"ID":"9c8b2164-2ab2-4a8f-9549-1ebf5a50800e","Type":"ContainerStarted","Data":"011aec443065412701899e6e99f544bb9606e89f9f69c064f73ea33066472644"}
Jul 09 16:51:13.965112 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.965084    2567 generic.go:356] "Generic (PLEG): container finished" podID="60d6296c-10c8-498f-b402-cfbf51a4f1b3" containerID="6f1e7ae2d44b2c8b01276a39b5af35f6b173caee3332be0a6d2ef57762b95156" exitCode=0
Jul 09 16:51:13.965692 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.965668    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerDied","Data":"6f1e7ae2d44b2c8b01276a39b5af35f6b173caee3332be0a6d2ef57762b95156"}
Jul 09 16:51:13.970188 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.970161    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2fhkn" event={"ID":"f4d2921f-10ef-49f7-a552-6e511cb4435f","Type":"ContainerStarted","Data":"b9b9491c1d6809a1558ac297fb13cfc99874e78aa4a9bd3aaf5eb25de35dda97"}
Jul 09 16:51:13.982575 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.982504    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" podStartSLOduration=12.982484877 podStartE2EDuration="12.982484877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:13.96802934 +0000 UTC m=+14.665201668" watchObservedRunningTime="2026-07-09 16:51:13.982484877 +0000 UTC m=+14.679657206"
Jul 09 16:51:13.982743 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:13.982718    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dqmmt" podStartSLOduration=3.895568478 podStartE2EDuration="14.982707818s" totalImagesPullingTime="11.08713934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:13.982355904 +0000 UTC m=+14.679528233" watchObservedRunningTime="2026-07-09 16:51:13.982707818 +0000 UTC m=+14.679880147"
Jul 09 16:51:14.012124 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:14.012064    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5jqxb" podStartSLOduration=3.925392894 podStartE2EDuration="15.012042579s" totalImagesPullingTime="11.086649685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:14.011507067 +0000 UTC m=+14.708679410" watchObservedRunningTime="2026-07-09 16:51:14.012042579 +0000 UTC m=+14.709214906"
Jul 09 16:51:14.012368 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:14.012315    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vp2jf" podStartSLOduration=3.968705373 podStartE2EDuration="15.012303677s" totalImagesPullingTime="11.043598304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:13.996029986 +0000 UTC m=+14.693202314" watchObservedRunningTime="2026-07-09 16:51:14.012303677 +0000 UTC m=+14.709476004"
Jul 09 16:51:14.051922 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:14.051853    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2fhkn" podStartSLOduration=4.278597156 podStartE2EDuration="15.051837147s" totalImagesPullingTime="10.773239991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:14.051603698 +0000 UTC m=+14.748776025" watchObservedRunningTime="2026-07-09 16:51:14.051837147 +0000 UTC m=+14.749009474"
Jul 09 16:51:14.866222 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:14.866178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:14.866496 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:14.866299    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:14.974459 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:14.974348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xnpm5" event={"ID":"8b308928-5e0e-4beb-aa2d-1f4fe94348e5","Type":"ContainerStarted","Data":"1495d9cf1aef667100e364dcf3ca12b4c251e6ba89a670478e96026fc3270665"}
Jul 09 16:51:14.990247 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:14.990185    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xnpm5" podStartSLOduration=4.876536429 podStartE2EDuration="15.990166713s" totalImagesPullingTime="11.113630284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:14.989966525 +0000 UTC m=+15.687138853" watchObservedRunningTime="2026-07-09 16:51:14.990166713 +0000 UTC m=+15.687339053"
Jul 09 16:51:15.867156 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:15.867117    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:15.867318 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:15.867118    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:15.867318 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:15.867255    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:15.867400 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:15.867360    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:16.866546 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:16.866503    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:16.867174 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:16.866633    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:17.286185 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.286113    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:17.286924 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.286902    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:17.444770 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.444726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:17.444770 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.444775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:17.445137 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.444902    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:17.445137 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.444902    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:17.445137 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.444987    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:33.444965749 +0000 UTC m=+34.142138056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:17.445137 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.445094    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:33.445064568 +0000 UTC m=+34.142236903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:17.545318 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.545228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:17.545478 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.545419    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:17.545478 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.545444    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:17.545478 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.545456    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:17.545619 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.545515    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:33.545494979 +0000 UTC m=+34.242667286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:17.866641 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.866554    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:17.867115 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.866702    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:17.867115 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.866758    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:17.867115 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:17.866887    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:17.980860 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.980831    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:17.981303 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:17.981287    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vp2jf"
Jul 09 16:51:18.867220 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:18.866995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:18.867686 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:18.867328    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:19.867379 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:19.867349    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:19.867857 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:19.867384    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:19.867857 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:19.867449    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:19.867857 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:19.867605    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:20.866629 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:20.866594    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:20.866823 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:20.866720    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:21.866957 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:21.866926    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:21.867537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:21.866927    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:21.867537 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:21.867071    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:21.867537 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:21.867097    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:22.866701 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:22.866669    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:22.866894 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:22.866777    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:23.866814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:23.866774    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:23.867278 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:23.866790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:23.867278 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:23.866947    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:23.867278 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:23.867015    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:24.757246 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.757222    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 16:51:24.774050 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.773943    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T16:51:24.75724113Z","UUID":"cdc4e148-b5a1-46b7-a967-7a9ddd58a8b8","Handler":null,"Name":"","Endpoint":""}
Jul 09 16:51:24.775293 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.775273    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 16:51:24.775379 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.775320    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 16:51:24.866502 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.866466    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:24.866666 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:24.866595    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:24.996709 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.996513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl" event={"ID":"9260de68-1f8b-475a-a8af-c8cb571c38f7","Type":"ContainerStarted","Data":"546871dc776bf9d7be70a9b58573a1eb9533892092b2d5c3f5e4b87f7c755c85"}
Jul 09 16:51:24.998216 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.998196    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"a4eebbb499e85f46114f673572ba273b43af344c5dac2750104dc8ace2ba3897"}
Jul 09 16:51:24.998307 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.998252    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"d224d2b388d724c32fc18cd0f9c6f2398503845eff40fa5142a41da9f4e239ee"}
Jul 09 16:51:24.999524 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:24.999500    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fdgf" event={"ID":"6bc945b8-7b27-42e2-b286-4020fdbd916f","Type":"ContainerStarted","Data":"6662ba2867dc3aebb4856c6bde44623ebd36f292bdab01d6a4d883fb1e20f707"}
Jul 09 16:51:25.019585 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:25.019540    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9fdgf" podStartSLOduration=4.978068753 podStartE2EDuration="26.019524865s" totalImagesPullingTime="21.041456112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:25.019261983 +0000 UTC m=+25.716434309" watchObservedRunningTime="2026-07-09 16:51:25.019524865 +0000 UTC m=+25.716697193"
Jul 09 16:51:25.867111 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:25.867081    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:25.867311 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:25.867091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:25.867311 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:25.867187    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:25.867311 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:25.867267    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:26.003762 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.003675    2567 generic.go:356] "Generic (PLEG): container finished" podID="60d6296c-10c8-498f-b402-cfbf51a4f1b3" containerID="f255c7f85a8bed87076287ee34fa7efe350beccf9014fbef3531ce304ad14705" exitCode=0
Jul 09 16:51:26.004344 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.003762    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerDied","Data":"f255c7f85a8bed87076287ee34fa7efe350beccf9014fbef3531ce304ad14705"}
Jul 09 16:51:26.006876 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.006846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"70d6ddac739cce7d67e048a1f0d5f2a33fdb251fd7c07ab265b0799a7b08abc2"}
Jul 09 16:51:26.006998 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.006939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"84445da7700c41e17e020e86f8ed58933e8fbb60dd4a76078b9239dc9dd235b1"}
Jul 09 16:51:26.006998 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.006993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"da4176c4f07db34033dfb9d5ab7f51a5ddd08efc64fb9495a3c291914ec312b8"}
Jul 09 16:51:26.007114 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.007036    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"1f98a229f9f693f4f0d6346a770f7376ec965a5d49cc832b816049aebb106974"}
Jul 09 16:51:26.866690 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:26.866655    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:26.866995 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:26.866806    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:27.305140 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:27.305118    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqmmt_9c8b2164-2ab2-4a8f-9549-1ebf5a50800e/dns-node-resolver/0.log"
Jul 09 16:51:27.867140 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:27.867050    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:27.867140 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:27.867098    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:27.867331 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:27.867189    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:27.867331 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:27.867293    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:28.013898 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.013863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"a8646769aeaa006662b6ce1c9b3d6633451e315f100b32af2a446d9943f3cb11"}
Jul 09 16:51:28.015904 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.015880    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl" event={"ID":"9260de68-1f8b-475a-a8af-c8cb571c38f7","Type":"ContainerStarted","Data":"539b17e11e6892203c3672e5a5bdbd298c482f997c85227571dfd3793594b326"}
Jul 09 16:51:28.018010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.017988    2567 generic.go:356] "Generic (PLEG): container finished" podID="60d6296c-10c8-498f-b402-cfbf51a4f1b3" containerID="9e50af1f2394c45909c0e4c626cbdd9b314942e5dae7f9cfea63f93220c4352c" exitCode=0
Jul 09 16:51:28.018121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.018017    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerDied","Data":"9e50af1f2394c45909c0e4c626cbdd9b314942e5dae7f9cfea63f93220c4352c"}
Jul 09 16:51:28.035304 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.035256    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbfzl" podStartSLOduration=5.745836669 podStartE2EDuration="29.035240883s" totalImagesPullingTime="23.289404214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:28.034958784 +0000 UTC m=+28.732131110" watchObservedRunningTime="2026-07-09 16:51:28.035240883 +0000 UTC m=+28.732413209"
Jul 09 16:51:28.287686 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.287617    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2fhkn_f4d2921f-10ef-49f7-a552-6e511cb4435f/node-ca/0.log"
Jul 09 16:51:28.866765 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:28.866735    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:28.867175 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:28.866851    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:29.870149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:29.869922    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:29.870149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:29.869973    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:29.870630 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:29.870168    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:29.870630 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:29.870246    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:30.025848 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.025790    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" event={"ID":"3afc9352-6013-476b-97ed-7e2834837109","Type":"ContainerStarted","Data":"d8218081c1f3b373f45a2fafbcbbb50182c1949e8382eab3fb6e04f021d2fb85"}
Jul 09 16:51:30.026116 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.026095    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:30.026189 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.026169    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:30.026236 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.026219    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:30.043930 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.043906    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:30.044059 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.044011    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:51:30.056729 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.056682    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd" podStartSLOduration=8.449324465 podStartE2EDuration="31.056667785s" totalImagesPullingTime="22.60734332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:30.056045646 +0000 UTC m=+30.753217988" watchObservedRunningTime="2026-07-09 16:51:30.056667785 +0000 UTC m=+30.753840118"
Jul 09 16:51:30.866698 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:30.866657    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:30.866865 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:30.866772    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:31.030957 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:31.030864    2567 generic.go:356] "Generic (PLEG): container finished" podID="60d6296c-10c8-498f-b402-cfbf51a4f1b3" containerID="df9c7a516d2aa9bac57691e85987c14ae7c7a2bc3e3153cba6bd655fbfda1f0d" exitCode=0
Jul 09 16:51:31.030957 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:31.030913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerDied","Data":"df9c7a516d2aa9bac57691e85987c14ae7c7a2bc3e3153cba6bd655fbfda1f0d"}
Jul 09 16:51:31.866902 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:31.866861    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:31.867076 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:31.866861    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:31.867076 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:31.867002    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:31.867076 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:31.867063    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:32.866356 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:32.866316    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:32.866849 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:32.866425    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:33.455511 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:33.455475    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:33.455511 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:33.455515    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:33.455755 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.455609    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:33.455755 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.455619    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:33.455755 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.455662    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret podName:77fbfceb-30f2-4e00-9a5f-c083f062c845 nodeName:}" failed. No retries permitted until 2026-07-09 16:52:05.455648641 +0000 UTC m=+66.152820947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret") pod "global-pull-secret-syncer-4fx9l" (UID: "77fbfceb-30f2-4e00-9a5f-c083f062c845") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:33.455755 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.455676    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs podName:2cff268e-e6cc-4a37-bac8-b79cd442a131 nodeName:}" failed. No retries permitted until 2026-07-09 16:52:05.45566908 +0000 UTC m=+66.152841385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs") pod "network-metrics-daemon-884hx" (UID: "2cff268e-e6cc-4a37-bac8-b79cd442a131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:33.556711 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:33.556670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:33.556926 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.556885    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:33.556926 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.556904    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:33.556926 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.556916    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lpkzz for pod openshift-network-diagnostics/network-check-target-bh5ql: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:33.557036 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.556975    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz podName:1bdb1bd7-4822-45f9-a873-a819a53b6042 nodeName:}" failed. No retries permitted until 2026-07-09 16:52:05.556960474 +0000 UTC m=+66.254132778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpkzz" (UniqueName: "kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz") pod "network-check-target-bh5ql" (UID: "1bdb1bd7-4822-45f9-a873-a819a53b6042") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:33.866878 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:33.866779    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:33.866878 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:33.866821    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:33.867433 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.866945    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:33.867433 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:33.867032    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:34.866451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:34.866415    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:34.866638 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:34.866549    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:35.866247 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:35.866211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:35.866709 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:35.866213    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:35.866709 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:35.866349    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:35.866709 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:35.866443    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:36.866531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:36.866491    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:36.866926 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:36.866615    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:37.050747 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:37.050654    2567 generic.go:356] "Generic (PLEG): container finished" podID="60d6296c-10c8-498f-b402-cfbf51a4f1b3" containerID="c9d19a15db6bc4144e3dcec28580382f4c5ec8cdc6ca466611aca17124d46918" exitCode=0
Jul 09 16:51:37.050747 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:37.050703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerDied","Data":"c9d19a15db6bc4144e3dcec28580382f4c5ec8cdc6ca466611aca17124d46918"}
Jul 09 16:51:37.866733 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:37.866691    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:37.867243 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:37.866837    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:37.867243 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:37.866885    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:37.867243 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:37.866970    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:38.055778 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:38.055740    2567 generic.go:356] "Generic (PLEG): container finished" podID="60d6296c-10c8-498f-b402-cfbf51a4f1b3" containerID="3e8869a55eee76c3b3a161d898839a2618275e6780430d6c6098e9c12108187a" exitCode=0
Jul 09 16:51:38.055962 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:38.055836    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerDied","Data":"3e8869a55eee76c3b3a161d898839a2618275e6780430d6c6098e9c12108187a"}
Jul 09 16:51:38.867233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:38.867037    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:38.867628 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:38.867306    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:39.062042 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:39.062000    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brnqb" event={"ID":"60d6296c-10c8-498f-b402-cfbf51a4f1b3","Type":"ContainerStarted","Data":"1e8f6193e95e9865a7e9ab7bb6f1f33305f7719e12e19ad2f240ffe0c82f4a1b"}
Jul 09 16:51:39.088274 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:39.088224    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-brnqb" podStartSLOduration=8.399219852 podStartE2EDuration="40.088209034s" totalImagesPullingTime="31.688989182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:39.087857411 +0000 UTC m=+39.785029737" watchObservedRunningTime="2026-07-09 16:51:39.088209034 +0000 UTC m=+39.785381360"
Jul 09 16:51:39.867567 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:39.867526    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:39.868030 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:39.867634    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:39.868030 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:39.867672    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:39.868030 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:39.867776    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:40.866557 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:40.866525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:40.866734 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:40.866634    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:41.866664 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:41.866625    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:41.867094 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:41.866745    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:41.867094 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:41.866791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:41.867094 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:41.866908    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:42.867113 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:42.867074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:42.867497 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:42.867192    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:43.866351 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:43.866311    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:43.866553 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:43.866311    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:43.866553 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:43.866454    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:43.866553 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:43.866493    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:44.866715 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:44.866679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:44.867124 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:44.866783    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:45.866682 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:45.866653    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:45.866883 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:45.866692    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:45.866883 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:45.866762    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:45.867190 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:45.866885    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:46.867020 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:46.866986    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:46.867406 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:46.867085    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:47.866666 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:47.866632    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:47.866915 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:47.866737    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:47.866915 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:47.866791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:47.866915 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:47.866891    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:48.866302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:48.866264    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:48.866704 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:48.866380    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:49.871122 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:49.871096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:49.871858 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:49.871241    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:49.871858 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:49.871289    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:49.871858 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:49.871399    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:49.883912 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:49.883775    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fx9l"]
Jul 09 16:51:49.884292 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:49.884271    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:49.884421 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:49.884397    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:49.884655 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:49.884641    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-884hx"]
Jul 09 16:51:49.885226 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:49.885167    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bh5ql"]
Jul 09 16:51:50.086495 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:50.086415    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:50.086640 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:50.086416    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:50.086640 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:50.086533    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:50.086640 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:50.086617    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:51.866507 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:51.866473    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:51.867036 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:51.866484    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:51.867036 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:51.866596    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:51.867036 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:51.866494    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:51.867036 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:51.866658    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:51.867036 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:51.866771    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:53.866855 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:53.866644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:53.867316 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:53.866644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:53.867316 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:53.866925    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bh5ql" podUID="1bdb1bd7-4822-45f9-a873-a819a53b6042"
Jul 09 16:51:53.867316 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:53.867011    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-884hx" podUID="2cff268e-e6cc-4a37-bac8-b79cd442a131"
Jul 09 16:51:53.867316 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:53.866667    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:53.867316 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:51:53.867085    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fx9l" podUID="77fbfceb-30f2-4e00-9a5f-c083f062c845"
Jul 09 16:51:54.169647 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.169569    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeReady"
Jul 09 16:51:54.169789 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.169691    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 16:51:54.230854 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.230822    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ccp7t"]
Jul 09 16:51:54.232659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.232644    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jgqv6"]
Jul 09 16:51:54.232780 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.232761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.234452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.234432    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.235659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.235635    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 16:51:54.235771 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.235661    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 16:51:54.236014 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.235995    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pdg8n\""
Jul 09 16:51:54.237853 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.237834    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 16:51:54.237945 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.237893    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 16:51:54.238178 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.238163    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 16:51:54.238493 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.238477    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pbgfq\""
Jul 09 16:51:54.246777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.246759    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgqv6"]
Jul 09 16:51:54.251502 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.251482    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ccp7t"]
Jul 09 16:51:54.261649 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.261627    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ljgb9"]
Jul 09 16:51:54.263479 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.263462    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.268096 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.268069    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 16:51:54.268235 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.268187    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 16:51:54.268580 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.268558    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7gwx2\""
Jul 09 16:51:54.268580 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.268574    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 16:51:54.268723 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.268663    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 16:51:54.282103 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.282078    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ljgb9"]
Jul 09 16:51:54.404586 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404554    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7820052a-8a08-454a-9f39-c861fc961261-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.404586 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvh8\" (UniqueName: \"kubernetes.io/projected/7820052a-8a08-454a-9f39-c861fc961261-kube-api-access-rnvh8\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7820052a-8a08-454a-9f39-c861fc961261-crio-socket\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404659    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22mp\" (UniqueName: \"kubernetes.io/projected/ce335e21-41ff-45cc-99ab-fa94472931be-kube-api-access-g22mp\") pod \"ingress-canary-jgqv6\" (UID: \"ce335e21-41ff-45cc-99ab-fa94472931be\") " pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404676    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7820052a-8a08-454a-9f39-c861fc961261-data-volume\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404732    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7820052a-8a08-454a-9f39-c861fc961261-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/156322e4-dcff-40e1-97d6-d11f5fc4c904-config-volume\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404776    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/156322e4-dcff-40e1-97d6-d11f5fc4c904-metrics-tls\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.404819 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156322e4-dcff-40e1-97d6-d11f5fc4c904-tmp-dir\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.405033 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce335e21-41ff-45cc-99ab-fa94472931be-cert\") pod \"ingress-canary-jgqv6\" (UID: \"ce335e21-41ff-45cc-99ab-fa94472931be\") " pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.405033 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.404860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhmp\" (UniqueName: \"kubernetes.io/projected/156322e4-dcff-40e1-97d6-d11f5fc4c904-kube-api-access-zkhmp\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.506004 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.505917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce335e21-41ff-45cc-99ab-fa94472931be-cert\") pod \"ingress-canary-jgqv6\" (UID: \"ce335e21-41ff-45cc-99ab-fa94472931be\") " pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.506004 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.505956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhmp\" (UniqueName: \"kubernetes.io/projected/156322e4-dcff-40e1-97d6-d11f5fc4c904-kube-api-access-zkhmp\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.506004 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.505979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7820052a-8a08-454a-9f39-c861fc961261-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvh8\" (UniqueName: \"kubernetes.io/projected/7820052a-8a08-454a-9f39-c861fc961261-kube-api-access-rnvh8\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506064    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7820052a-8a08-454a-9f39-c861fc961261-crio-socket\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506090    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22mp\" (UniqueName: \"kubernetes.io/projected/ce335e21-41ff-45cc-99ab-fa94472931be-kube-api-access-g22mp\") pod \"ingress-canary-jgqv6\" (UID: \"ce335e21-41ff-45cc-99ab-fa94472931be\") " pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506118    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7820052a-8a08-454a-9f39-c861fc961261-data-volume\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506172    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7820052a-8a08-454a-9f39-c861fc961261-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506194    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7820052a-8a08-454a-9f39-c861fc961261-crio-socket\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506201    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/156322e4-dcff-40e1-97d6-d11f5fc4c904-config-volume\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506248    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/156322e4-dcff-40e1-97d6-d11f5fc4c904-metrics-tls\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.506302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156322e4-dcff-40e1-97d6-d11f5fc4c904-tmp-dir\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.506666 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7820052a-8a08-454a-9f39-c861fc961261-data-volume\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506666 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506556    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156322e4-dcff-40e1-97d6-d11f5fc4c904-tmp-dir\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.506749 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506732    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7820052a-8a08-454a-9f39-c861fc961261-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.506953 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.506933    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/156322e4-dcff-40e1-97d6-d11f5fc4c904-config-volume\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.510390 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.510370    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7820052a-8a08-454a-9f39-c861fc961261-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.510492 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.510411    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/156322e4-dcff-40e1-97d6-d11f5fc4c904-metrics-tls\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.510492 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.510461    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce335e21-41ff-45cc-99ab-fa94472931be-cert\") pod \"ingress-canary-jgqv6\" (UID: \"ce335e21-41ff-45cc-99ab-fa94472931be\") " pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.515630 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.515603    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhmp\" (UniqueName: \"kubernetes.io/projected/156322e4-dcff-40e1-97d6-d11f5fc4c904-kube-api-access-zkhmp\") pod \"dns-default-ccp7t\" (UID: \"156322e4-dcff-40e1-97d6-d11f5fc4c904\") " pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.518067 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.518041    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvh8\" (UniqueName: \"kubernetes.io/projected/7820052a-8a08-454a-9f39-c861fc961261-kube-api-access-rnvh8\") pod \"insights-runtime-extractor-ljgb9\" (UID: \"7820052a-8a08-454a-9f39-c861fc961261\") " pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.518149 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.518046    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22mp\" (UniqueName: \"kubernetes.io/projected/ce335e21-41ff-45cc-99ab-fa94472931be-kube-api-access-g22mp\") pod \"ingress-canary-jgqv6\" (UID: \"ce335e21-41ff-45cc-99ab-fa94472931be\") " pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.543996 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.543970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:54.549600 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.549580    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgqv6"
Jul 09 16:51:54.571834 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.571784    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ljgb9"
Jul 09 16:51:54.709169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.709136    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ccp7t"]
Jul 09 16:51:54.712738 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:54.712512    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156322e4_dcff_40e1_97d6_d11f5fc4c904.slice/crio-dc5c0fbdbd5e7181f4ea39e3c3d13665fd8bc65cc08d6387fba5d066a4a2d15e WatchSource:0}: Error finding container dc5c0fbdbd5e7181f4ea39e3c3d13665fd8bc65cc08d6387fba5d066a4a2d15e: Status 404 returned error can't find the container with id dc5c0fbdbd5e7181f4ea39e3c3d13665fd8bc65cc08d6387fba5d066a4a2d15e
Jul 09 16:51:54.713350 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.713324    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgqv6"]
Jul 09 16:51:54.716283 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:54.716263    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce335e21_41ff_45cc_99ab_fa94472931be.slice/crio-f83782e6dc7d7e40f765d5a454a478bdab2bbd13ff2095f4425baa3265382bf4 WatchSource:0}: Error finding container f83782e6dc7d7e40f765d5a454a478bdab2bbd13ff2095f4425baa3265382bf4: Status 404 returned error can't find the container with id f83782e6dc7d7e40f765d5a454a478bdab2bbd13ff2095f4425baa3265382bf4
Jul 09 16:51:54.728175 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:54.728153    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ljgb9"]
Jul 09 16:51:54.731697 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:51:54.731665    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7820052a_8a08_454a_9f39_c861fc961261.slice/crio-d8de4279642fb56f49ddf2675c608b75a469a2590cabed0d24e0dfffd1f13294 WatchSource:0}: Error finding container d8de4279642fb56f49ddf2675c608b75a469a2590cabed0d24e0dfffd1f13294: Status 404 returned error can't find the container with id d8de4279642fb56f49ddf2675c608b75a469a2590cabed0d24e0dfffd1f13294
Jul 09 16:51:55.099246 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.099207    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccp7t" event={"ID":"156322e4-dcff-40e1-97d6-d11f5fc4c904","Type":"ContainerStarted","Data":"dc5c0fbdbd5e7181f4ea39e3c3d13665fd8bc65cc08d6387fba5d066a4a2d15e"}
Jul 09 16:51:55.100452 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.100423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ljgb9" event={"ID":"7820052a-8a08-454a-9f39-c861fc961261","Type":"ContainerStarted","Data":"d8de4279642fb56f49ddf2675c608b75a469a2590cabed0d24e0dfffd1f13294"}
Jul 09 16:51:55.101576 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.101550    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgqv6" event={"ID":"ce335e21-41ff-45cc-99ab-fa94472931be","Type":"ContainerStarted","Data":"f83782e6dc7d7e40f765d5a454a478bdab2bbd13ff2095f4425baa3265382bf4"}
Jul 09 16:51:55.866514 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.866474    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:51:55.866685 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.866477    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:51:55.866880 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.866477    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:51:55.870745 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.869489    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 16:51:55.871814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.871247    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k87dn\""
Jul 09 16:51:55.871814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.871265    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4z5xv\""
Jul 09 16:51:55.871814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.871335    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 16:51:55.871814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.871381    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 16:51:55.871814 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:55.871593    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 16:51:57.109261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:57.109227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccp7t" event={"ID":"156322e4-dcff-40e1-97d6-d11f5fc4c904","Type":"ContainerStarted","Data":"d22a1d872f065722cd58d8164ebe87d9ced1a35027cc27056dda8481137ebf6e"}
Jul 09 16:51:58.113699 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:58.113659    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgqv6" event={"ID":"ce335e21-41ff-45cc-99ab-fa94472931be","Type":"ContainerStarted","Data":"203004087d84e00b77447e9df79d89820dab108fc6d2877e84b3e84630d0f175"}
Jul 09 16:51:58.115453 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:58.115428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccp7t" event={"ID":"156322e4-dcff-40e1-97d6-d11f5fc4c904","Type":"ContainerStarted","Data":"b68d46e4a3aa199fc8a45a37faf78ec4fb3a662c940cdd54ebf8abfec4720683"}
Jul 09 16:51:58.115575 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:58.115553    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:51:58.129786 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:58.129746    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jgqv6" podStartSLOduration=1.199861602 podStartE2EDuration="4.129734155s" totalImagesPullingTime="2.929872553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:58.129130278 +0000 UTC m=+58.826302603" watchObservedRunningTime="2026-07-09 16:51:58.129734155 +0000 UTC m=+58.826906483"
Jul 09 16:51:58.145596 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:51:58.145559    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ccp7t" podStartSLOduration=2.34130648 podStartE2EDuration="4.145545582s" totalImagesPullingTime="1.804239102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:58.145176755 +0000 UTC m=+58.842349106" watchObservedRunningTime="2026-07-09 16:51:58.145545582 +0000 UTC m=+58.842717906"
Jul 09 16:52:00.860428 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.860387    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f7594b87-wp8zq"]
Jul 09 16:52:00.871758 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.871722    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.874502 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.874472    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f7594b87-wp8zq"]
Jul 09 16:52:00.874990 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.874963    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 16:52:00.875135 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875076    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 16:52:00.875135 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875095    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 16:52:00.875253 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875196    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tnbw7\""
Jul 09 16:52:00.875391 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875370    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 16:52:00.875461 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875405    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 16:52:00.875522 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875491    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 16:52:00.875646 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.875629    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 16:52:00.882011 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.881758    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 16:52:00.957123 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-serving-cert\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.957123 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4bn\" (UniqueName: \"kubernetes.io/projected/da48f9a4-3950-42c9-9228-017301d59839-kube-api-access-4b4bn\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.957382 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957162    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-oauth-config\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.957546 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-console-config\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.957685 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957599    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-trusted-ca-bundle\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.957685 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-service-ca\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:00.957788 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:00.957705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-oauth-serving-cert\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058263 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-serving-cert\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058263 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4bn\" (UniqueName: \"kubernetes.io/projected/da48f9a4-3950-42c9-9228-017301d59839-kube-api-access-4b4bn\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058517 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-oauth-config\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058517 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-console-config\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058517 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-trusted-ca-bundle\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058667 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-service-ca\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.058667 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.058596    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-oauth-serving-cert\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.059436 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.059402    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-service-ca\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.059616 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.059597    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-trusted-ca-bundle\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.059705 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.059648    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-oauth-serving-cert\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.059705 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.059653    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-console-config\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.060963 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.060939    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-oauth-config\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.061169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.061148    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-serving-cert\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.067356 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.067332    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4bn\" (UniqueName: \"kubernetes.io/projected/da48f9a4-3950-42c9-9228-017301d59839-kube-api-access-4b4bn\") pod \"console-57f7594b87-wp8zq\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") " pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:01.183983 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:01.183856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:02.045167 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.045135    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ffwd"
Jul 09 16:52:02.296555 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.296478    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f7594b87-wp8zq"]
Jul 09 16:52:02.301553 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:02.301522    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda48f9a4_3950_42c9_9228_017301d59839.slice/crio-f71aad705046a2f54d306deae73c6b8b932eb447e5a3ca7ed885d1b6c1525fd9 WatchSource:0}: Error finding container f71aad705046a2f54d306deae73c6b8b932eb447e5a3ca7ed885d1b6c1525fd9: Status 404 returned error can't find the container with id f71aad705046a2f54d306deae73c6b8b932eb447e5a3ca7ed885d1b6c1525fd9
Jul 09 16:52:02.621415 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.621376    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"]
Jul 09 16:52:02.624568 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.624545    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.627262 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.627183    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 16:52:02.627262 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.627198    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 16:52:02.627262 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.627207    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sxczq\""
Jul 09 16:52:02.627262 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.627211    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 16:52:02.628258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.628244    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 16:52:02.628319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.628243    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 16:52:02.636852 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.636828    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"]
Jul 09 16:52:02.658875 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.658844    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-fk57g"]
Jul 09 16:52:02.661857 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.661837    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.664539 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.664518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 16:52:02.664675 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.664627    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 16:52:02.664675 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.664647    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h9p8r\""
Jul 09 16:52:02.664826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.664691    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 16:52:02.671418 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671397    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-94g6p"]
Jul 09 16:52:02.671641 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671622    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.671704 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.671743 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5mrx\" (UniqueName: \"kubernetes.io/projected/d2683169-3272-4fda-9422-bb4e082a59da-kube-api-access-f5mrx\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.671776 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt4w5\" (UniqueName: \"kubernetes.io/projected/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-api-access-xt4w5\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.671835 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671774    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.671835 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2683169-3272-4fda-9422-bb4e082a59da-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.671896 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.671928 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.671964 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.672010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.671994    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.676107 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.674844    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.678180 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.678159    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 16:52:02.678282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.678260    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-fk57g"]
Jul 09 16:52:02.679019 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.678994    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 16:52:02.679127 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.679044    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q92qp\""
Jul 09 16:52:02.679256 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.679002    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 16:52:02.773294 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773262    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2683169-3272-4fda-9422-bb4e082a59da-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.773459 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.773459 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-root\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.773459 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.773609 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-sys\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.773609 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.773609 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-textfile\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.773725 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.773725 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.773725 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-tls\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.773898 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f9e506-cc80-464f-9113-a8bbbffd1743-metrics-client-ca\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.773898 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.773898 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773857    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.773898 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773891    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-wtmp\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773929    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5mrx\" (UniqueName: \"kubernetes.io/projected/d2683169-3272-4fda-9422-bb4e082a59da-kube-api-access-f5mrx\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt4w5\" (UniqueName: \"kubernetes.io/projected/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-api-access-xt4w5\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.773981    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-accelerators-collector-config\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.774007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.774016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.774031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6tc\" (UniqueName: \"kubernetes.io/projected/43f9e506-cc80-464f-9113-a8bbbffd1743-kube-api-access-7b6tc\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.774026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2683169-3272-4fda-9422-bb4e082a59da-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.774098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.774213 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:02.774162    2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 09 16:52:02.774595 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:02.774234    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls podName:d2683169-3272-4fda-9422-bb4e082a59da nodeName:}" failed. No retries permitted until 2026-07-09 16:52:03.274216379 +0000 UTC m=+63.971388688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-gzpb4" (UID: "d2683169-3272-4fda-9422-bb4e082a59da") : secret "openshift-state-metrics-tls" not found
Jul 09 16:52:02.774595 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:02.774391    2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 09 16:52:02.774595 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:02.774461    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-tls podName:4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479 nodeName:}" failed. No retries permitted until 2026-07-09 16:52:03.274442965 +0000 UTC m=+63.971615270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-fk57g" (UID: "4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479") : secret "kube-state-metrics-tls" not found
Jul 09 16:52:02.774916 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.774854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.776262 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.776240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.776542 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.776521    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.784013 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.783991    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5mrx\" (UniqueName: \"kubernetes.io/projected/d2683169-3272-4fda-9422-bb4e082a59da-kube-api-access-f5mrx\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:02.784346 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.784329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt4w5\" (UniqueName: \"kubernetes.io/projected/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-api-access-xt4w5\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:02.875347 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-tls\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875457 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f9e506-cc80-464f-9113-a8bbbffd1743-metrics-client-ca\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875457 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-wtmp\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875556 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-accelerators-collector-config\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875556 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6tc\" (UniqueName: \"kubernetes.io/projected/43f9e506-cc80-464f-9113-a8bbbffd1743-kube-api-access-7b6tc\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875556 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-root\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875718 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-sys\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875718 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-textfile\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875718 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875718 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-wtmp\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875718 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875694    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-sys\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.875979 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43f9e506-cc80-464f-9113-a8bbbffd1743-root\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.876032 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.875995    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-textfile\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.876093 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.876038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f9e506-cc80-464f-9113-a8bbbffd1743-metrics-client-ca\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.876268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.876242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-accelerators-collector-config\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.878121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.878099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-tls\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.878272 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.878247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43f9e506-cc80-464f-9113-a8bbbffd1743-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.885556 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.885533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6tc\" (UniqueName: \"kubernetes.io/projected/43f9e506-cc80-464f-9113-a8bbbffd1743-kube-api-access-7b6tc\") pod \"node-exporter-94g6p\" (UID: \"43f9e506-cc80-464f-9113-a8bbbffd1743\") " pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.986382 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:02.986345    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-94g6p"
Jul 09 16:52:02.996910 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:02.996882    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f9e506_cc80_464f_9113_a8bbbffd1743.slice/crio-e3e36db4acaf54e58bca4dbf97d4e788508a9076b28beca818633b1acdf3fa77 WatchSource:0}: Error finding container e3e36db4acaf54e58bca4dbf97d4e788508a9076b28beca818633b1acdf3fa77: Status 404 returned error can't find the container with id e3e36db4acaf54e58bca4dbf97d4e788508a9076b28beca818633b1acdf3fa77
Jul 09 16:52:03.131570 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.131478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ljgb9" event={"ID":"7820052a-8a08-454a-9f39-c861fc961261","Type":"ContainerStarted","Data":"56becd0ecf06103e9194eaba1087b0b72a720220c64db94ee2b370c9cf7fe8e9"}
Jul 09 16:52:03.132052 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.131567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ljgb9" event={"ID":"7820052a-8a08-454a-9f39-c861fc961261","Type":"ContainerStarted","Data":"dd139a4c85556d6a57905248d6262fdc0c24d85cfbb7f262ae14b3838a97547c"}
Jul 09 16:52:03.132864 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.132828    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f7594b87-wp8zq" event={"ID":"da48f9a4-3950-42c9-9228-017301d59839","Type":"ContainerStarted","Data":"f71aad705046a2f54d306deae73c6b8b932eb447e5a3ca7ed885d1b6c1525fd9"}
Jul 09 16:52:03.134081 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.134056    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94g6p" event={"ID":"43f9e506-cc80-464f-9113-a8bbbffd1743","Type":"ContainerStarted","Data":"e3e36db4acaf54e58bca4dbf97d4e788508a9076b28beca818633b1acdf3fa77"}
Jul 09 16:52:03.279524 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.279480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:03.279721 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.279575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:03.279721 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:03.279660    2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 09 16:52:03.279877 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:03.279754    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls podName:d2683169-3272-4fda-9422-bb4e082a59da nodeName:}" failed. No retries permitted until 2026-07-09 16:52:04.27973236 +0000 UTC m=+64.976904667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-gzpb4" (UID: "d2683169-3272-4fda-9422-bb4e082a59da") : secret "openshift-state-metrics-tls" not found
Jul 09 16:52:03.282540 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.282512    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fk57g\" (UID: \"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:03.570533 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.570453    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g"
Jul 09 16:52:03.681828 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.681393    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 16:52:03.687047 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.685763    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.689152 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.688830    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 16:52:03.689152 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.688851    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 16:52:03.689152 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.688879    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 16:52:03.689152 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.689064    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 16:52:03.689613 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.689595    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 16:52:03.689756 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.689663    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4prsk\""
Jul 09 16:52:03.689916 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.689884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 16:52:03.690107 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.690091    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 16:52:03.690359 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.690344    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 16:52:03.690577 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.690544    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 16:52:03.695258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.695173    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 16:52:03.731074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.731037    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-fk57g"]
Jul 09 16:52:03.735068 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:03.735034    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5c1fd2_e143_4c4d_9dbd_3e9ecdd08479.slice/crio-6af9dd4a42ceb8a378bbd428b3fc57b312a7cd0b115acfec8439970013fbcaf8 WatchSource:0}: Error finding container 6af9dd4a42ceb8a378bbd428b3fc57b312a7cd0b115acfec8439970013fbcaf8: Status 404 returned error can't find the container with id 6af9dd4a42ceb8a378bbd428b3fc57b312a7cd0b115acfec8439970013fbcaf8
Jul 09 16:52:03.784639 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsdk\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-kube-api-access-qcsdk\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.784832 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784666    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.784832 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.784832 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784732    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.784832 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784820    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-volume\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785046 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785046 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784857    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785046 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784884    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785046 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784914    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785046 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.784991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785046 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.785027    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-web-config\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785293 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.785065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-out\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.785293 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.785089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886106 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886064    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsdk\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-kube-api-access-qcsdk\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886309 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886309 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886416 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886307    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886416 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-volume\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886416 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886552 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886552 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886552 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886697 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886570    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886697 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-web-config\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886697 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-out\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886697 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886915 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.886839    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.886973 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:03.886943    2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 09 16:52:03.887022 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:52:03.887002    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls podName:aede80d7-03eb-43cd-ac80-cf585225cc8e nodeName:}" failed. No retries permitted until 2026-07-09 16:52:04.386983399 +0000 UTC m=+65.084155718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e") : secret "alertmanager-main-tls" not found
Jul 09 16:52:03.887154 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.887041    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.888094 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.888070    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.889612 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.889583    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.889746 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.889723    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.890566 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.890088    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-volume\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.890566 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.890523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-web-config\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.890762 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.890734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.891016 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.890990    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-out\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.891136 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.891118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.891406 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.891382    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:03.903373 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:03.903347    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsdk\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-kube-api-access-qcsdk\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:04.139354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.139241    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g" event={"ID":"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479","Type":"ContainerStarted","Data":"6af9dd4a42ceb8a378bbd428b3fc57b312a7cd0b115acfec8439970013fbcaf8"}
Jul 09 16:52:04.289536 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.289505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:04.292200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.292180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2683169-3272-4fda-9422-bb4e082a59da-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-gzpb4\" (UID: \"d2683169-3272-4fda-9422-bb4e082a59da\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:04.390892 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.390809    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:04.393492 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.393456    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:04.434138 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.434026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"
Jul 09 16:52:04.581437 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.581399    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d56995d65-8sxml"]
Jul 09 16:52:04.585402 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.585365    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.588102 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.588072    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 16:52:04.588245 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.588201    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5xs2d\""
Jul 09 16:52:04.588313 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.588240    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 16:52:04.588391 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.588356    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 16:52:04.588459 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.588361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6ktmh43uvu698\""
Jul 09 16:52:04.588459 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.588422    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 16:52:04.589183 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.589167    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 16:52:04.598773 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.598748    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d56995d65-8sxml"]
Jul 09 16:52:04.600586 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.600555    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:52:04.693636 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.693636 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693596    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.693636 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.693918 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xdc\" (UniqueName: \"kubernetes.io/projected/87a7410d-336a-4e97-9507-f7432751a933-kube-api-access-h2xdc\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.693918 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-grpc-tls\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.693918 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a7410d-336a-4e97-9507-f7432751a933-metrics-client-ca\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.693918 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-tls\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.694079 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.693915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794618    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xdc\" (UniqueName: \"kubernetes.io/projected/87a7410d-336a-4e97-9507-f7432751a933-kube-api-access-h2xdc\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-grpc-tls\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a7410d-336a-4e97-9507-f7432751a933-metrics-client-ca\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-tls\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795130 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.794766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.795974 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.795913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a7410d-336a-4e97-9507-f7432751a933-metrics-client-ca\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.798826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.797585    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.798826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.797737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.798826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.798201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.798826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.798202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-tls\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.798826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.798308    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-grpc-tls\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.798826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.798313    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87a7410d-336a-4e97-9507-f7432751a933-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.803454 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.803429    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xdc\" (UniqueName: \"kubernetes.io/projected/87a7410d-336a-4e97-9507-f7432751a933-kube-api-access-h2xdc\") pod \"thanos-querier-6d56995d65-8sxml\" (UID: \"87a7410d-336a-4e97-9507-f7432751a933\") " pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:04.895765 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:04.895725    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:05.145510 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.145462    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94g6p" event={"ID":"43f9e506-cc80-464f-9113-a8bbbffd1743","Type":"ContainerStarted","Data":"aac6ae9fc5a7e091455a96bbe4e0b25bb10b1fb855fe62275a2f65dc565920a0"}
Jul 09 16:52:05.502341 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.502291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:52:05.502472 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.502375    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:52:05.505163 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.505139    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 16:52:05.505302 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.505139    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 16:52:05.515998 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.515972    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cff268e-e6cc-4a37-bac8-b79cd442a131-metrics-certs\") pod \"network-metrics-daemon-884hx\" (UID: \"2cff268e-e6cc-4a37-bac8-b79cd442a131\") " pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:52:05.516166 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.516146    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77fbfceb-30f2-4e00-9a5f-c083f062c845-original-pull-secret\") pod \"global-pull-secret-syncer-4fx9l\" (UID: \"77fbfceb-30f2-4e00-9a5f-c083f062c845\") " pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:52:05.603371 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.603339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:52:05.606114 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.606092    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 16:52:05.616696 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.616656    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 16:52:05.628850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.628770    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkzz\" (UniqueName: \"kubernetes.io/projected/1bdb1bd7-4822-45f9-a873-a819a53b6042-kube-api-access-lpkzz\") pod \"network-check-target-bh5ql\" (UID: \"1bdb1bd7-4822-45f9-a873-a819a53b6042\") " pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:52:05.644767 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.644676    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d56995d65-8sxml"]
Jul 09 16:52:05.649835 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:05.649787    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a7410d_336a_4e97_9507_f7432751a933.slice/crio-f87498270e5c2e044f772113a3b18f693b07958d839148dd8d10bdc8c16917fb WatchSource:0}: Error finding container f87498270e5c2e044f772113a3b18f693b07958d839148dd8d10bdc8c16917fb: Status 404 returned error can't find the container with id f87498270e5c2e044f772113a3b18f693b07958d839148dd8d10bdc8c16917fb
Jul 09 16:52:05.781426 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.781347    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fx9l"
Jul 09 16:52:05.792207 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.792179    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k87dn\""
Jul 09 16:52:05.798984 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.798952    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4z5xv\""
Jul 09 16:52:05.801049 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.800856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-884hx"
Jul 09 16:52:05.807402 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.807228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:52:05.890877 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.880311    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4"]
Jul 09 16:52:05.890877 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.884853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 16:52:05.898657 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:05.898120    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2683169_3272_4fda_9422_bb4e082a59da.slice/crio-23148e60b64e21624c858a2f82af70446c7f9a076a6b6519c348c7df72e21c07 WatchSource:0}: Error finding container 23148e60b64e21624c858a2f82af70446c7f9a076a6b6519c348c7df72e21c07: Status 404 returned error can't find the container with id 23148e60b64e21624c858a2f82af70446c7f9a076a6b6519c348c7df72e21c07
Jul 09 16:52:05.970376 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.970305    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fx9l"]
Jul 09 16:52:05.977651 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:05.977594    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fbfceb_30f2_4e00_9a5f_c083f062c845.slice/crio-9116a9fb60a0beb63996ce599c5219f5b13f2d7bd5c05be35e08b600ea0803ff WatchSource:0}: Error finding container 9116a9fb60a0beb63996ce599c5219f5b13f2d7bd5c05be35e08b600ea0803ff: Status 404 returned error can't find the container with id 9116a9fb60a0beb63996ce599c5219f5b13f2d7bd5c05be35e08b600ea0803ff
Jul 09 16:52:05.996284 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:05.996253    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bh5ql"]
Jul 09 16:52:06.002960 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:06.002911    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdb1bd7_4822_45f9_a873_a819a53b6042.slice/crio-0de33e5f4efd7861138cfe786de66357737ff0558500f7ed48032dc56251e00c WatchSource:0}: Error finding container 0de33e5f4efd7861138cfe786de66357737ff0558500f7ed48032dc56251e00c: Status 404 returned error can't find the container with id 0de33e5f4efd7861138cfe786de66357737ff0558500f7ed48032dc56251e00c
Jul 09 16:52:06.019330 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.019301    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-884hx"]
Jul 09 16:52:06.022576 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:52:06.022532    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cff268e_e6cc_4a37_bac8_b79cd442a131.slice/crio-a0dfd30af8a48349f14d40c5ea96cc59a0f520a89d5d97afe5ec9e72d17b4788 WatchSource:0}: Error finding container a0dfd30af8a48349f14d40c5ea96cc59a0f520a89d5d97afe5ec9e72d17b4788: Status 404 returned error can't find the container with id a0dfd30af8a48349f14d40c5ea96cc59a0f520a89d5d97afe5ec9e72d17b4788
Jul 09 16:52:06.153099 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.152452    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f7594b87-wp8zq" event={"ID":"da48f9a4-3950-42c9-9228-017301d59839","Type":"ContainerStarted","Data":"5ff753344e0c2a4244045105dec2a670da17584db8a25ee09d1af0aa2454885c"}
Jul 09 16:52:06.155592 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.155532    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4" event={"ID":"d2683169-3272-4fda-9422-bb4e082a59da","Type":"ContainerStarted","Data":"9c3005b4f637e54cbd876842b88e335978f9bb1716d6d46a76b6d497df59a33a"}
Jul 09 16:52:06.155592 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.155579    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4" event={"ID":"d2683169-3272-4fda-9422-bb4e082a59da","Type":"ContainerStarted","Data":"9ddbef8519b8ae06facb2ae11611cceaef7ec1cd6ded7a01bff456266f515a7f"}
Jul 09 16:52:06.155820 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.155599    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4" event={"ID":"d2683169-3272-4fda-9422-bb4e082a59da","Type":"ContainerStarted","Data":"23148e60b64e21624c858a2f82af70446c7f9a076a6b6519c348c7df72e21c07"}
Jul 09 16:52:06.157770 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.157736    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"b9c567f8ec8815d21b3c211479fc6ecbb31ea943a350f0968eabe1391e11f446"}
Jul 09 16:52:06.159535 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.159460    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"f87498270e5c2e044f772113a3b18f693b07958d839148dd8d10bdc8c16917fb"}
Jul 09 16:52:06.162121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.162001    2567 generic.go:356] "Generic (PLEG): container finished" podID="43f9e506-cc80-464f-9113-a8bbbffd1743" containerID="aac6ae9fc5a7e091455a96bbe4e0b25bb10b1fb855fe62275a2f65dc565920a0" exitCode=0
Jul 09 16:52:06.162121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.162073    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94g6p" event={"ID":"43f9e506-cc80-464f-9113-a8bbbffd1743","Type":"ContainerDied","Data":"aac6ae9fc5a7e091455a96bbe4e0b25bb10b1fb855fe62275a2f65dc565920a0"}
Jul 09 16:52:06.165534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.165472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fx9l" event={"ID":"77fbfceb-30f2-4e00-9a5f-c083f062c845","Type":"ContainerStarted","Data":"9116a9fb60a0beb63996ce599c5219f5b13f2d7bd5c05be35e08b600ea0803ff"}
Jul 09 16:52:06.167196 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.167121    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bh5ql" event={"ID":"1bdb1bd7-4822-45f9-a873-a819a53b6042","Type":"ContainerStarted","Data":"0de33e5f4efd7861138cfe786de66357737ff0558500f7ed48032dc56251e00c"}
Jul 09 16:52:06.169347 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.169294    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-884hx" event={"ID":"2cff268e-e6cc-4a37-bac8-b79cd442a131","Type":"ContainerStarted","Data":"a0dfd30af8a48349f14d40c5ea96cc59a0f520a89d5d97afe5ec9e72d17b4788"}
Jul 09 16:52:06.189955 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.189619    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f7594b87-wp8zq" podStartSLOduration=2.934970814 podStartE2EDuration="6.189599445s" totalImagesPullingTime="3.254628631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:06.170442585 +0000 UTC m=+66.867614914" watchObservedRunningTime="2026-07-09 16:52:06.189599445 +0000 UTC m=+66.886771774"
Jul 09 16:52:06.910988 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.909831    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8665cddf46-kq8wl"]
Jul 09 16:52:06.914548 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.913960    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:06.919826 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.919649    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 16:52:06.920576 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.920069    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-t22pt\""
Jul 09 16:52:06.920576 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.920293    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 16:52:06.920576 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.920431    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 16:52:06.920576 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.920478    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ciohhlshmghp2\""
Jul 09 16:52:06.920846 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.920681    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 16:52:06.926243 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:06.925784    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8665cddf46-kq8wl"]
Jul 09 16:52:07.017300 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e3c04e-029e-44be-bc68-cae624d998cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.017479 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-client-ca-bundle\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.017479 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pfz\" (UniqueName: \"kubernetes.io/projected/27e3c04e-029e-44be-bc68-cae624d998cd-kube-api-access-z2pfz\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.017479 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-secret-metrics-server-tls\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.017479 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017434    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27e3c04e-029e-44be-bc68-cae624d998cd-audit-log\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.017479 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27e3c04e-029e-44be-bc68-cae624d998cd-metrics-server-audit-profiles\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.017862 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.017510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-secret-metrics-server-client-certs\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-secret-metrics-server-tls\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118543    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27e3c04e-029e-44be-bc68-cae624d998cd-audit-log\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118576    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27e3c04e-029e-44be-bc68-cae624d998cd-metrics-server-audit-profiles\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118627    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-secret-metrics-server-client-certs\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e3c04e-029e-44be-bc68-cae624d998cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-client-ca-bundle\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119169 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.118753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pfz\" (UniqueName: \"kubernetes.io/projected/27e3c04e-029e-44be-bc68-cae624d998cd-kube-api-access-z2pfz\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.119652 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.119201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27e3c04e-029e-44be-bc68-cae624d998cd-audit-log\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.120412 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.120051    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27e3c04e-029e-44be-bc68-cae624d998cd-metrics-server-audit-profiles\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.120412 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.120354    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e3c04e-029e-44be-bc68-cae624d998cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.123816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.123263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-secret-metrics-server-client-certs\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.123816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.123330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-secret-metrics-server-tls\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.123816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.123762    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27e3c04e-029e-44be-bc68-cae624d998cd-client-ca-bundle\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.127548 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.127503    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pfz\" (UniqueName: \"kubernetes.io/projected/27e3c04e-029e-44be-bc68-cae624d998cd-kube-api-access-z2pfz\") pod \"metrics-server-8665cddf46-kq8wl\" (UID: \"27e3c04e-029e-44be-bc68-cae624d998cd\") " pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:07.179537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.179453    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94g6p" event={"ID":"43f9e506-cc80-464f-9113-a8bbbffd1743","Type":"ContainerStarted","Data":"68b752132fc7519edaab3d3b5b9d6bc2ae703c7aeb47284ee2cae62d594519b6"}
Jul 09 16:52:07.179537 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.179516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-94g6p" event={"ID":"43f9e506-cc80-464f-9113-a8bbbffd1743","Type":"ContainerStarted","Data":"9ae99d34da069f7703d1f8c382da15725aaeb4e9d924bd635638052e353dcef2"}
Jul 09 16:52:07.197859 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.197787    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-94g6p" podStartSLOduration=3.872230836 podStartE2EDuration="5.197765221s" totalImagesPullingTime="1.244710194s" totalInitContainerRuntime="80.824191ms" isStatefulPod=true podCreationTimestamp="2026-07-09 16:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:07.196897275 +0000 UTC m=+67.894069606" watchObservedRunningTime="2026-07-09 16:52:07.197765221 +0000 UTC m=+67.894937549"
Jul 09 16:52:07.229376 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:07.229145    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:08.135529 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.133999    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ccp7t"
Jul 09 16:52:08.795351 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.795316    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:52:08.801115 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.801085    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.804177 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.804150    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 16:52:08.804327 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.804165    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 16:52:08.804524 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.804504    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 16:52:08.805163 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805139    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 16:52:08.805282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805266    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805617    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805732    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rtrff\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805895    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805969    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.805982    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.806191    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 16:52:08.806354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.806236    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 16:52:08.806742 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.806595    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 16:52:08.806742 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.806675    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 16:52:08.807374 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.806963    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3arht6r5pkgqc\""
Jul 09 16:52:08.808685 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.808487    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 16:52:08.817379 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.817357    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:52:08.842233 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842200    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842399 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842399 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ghk\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-kube-api-access-x7ghk\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842399 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842330    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842399 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842359    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842399 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842619 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842411    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842619 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842619 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842513    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842619 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842619 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842589    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842654    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842696    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.842850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.843078 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.843078 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.842893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944248    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944334    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944388    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944531    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ghk\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-kube-api-access-x7ghk\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.945197 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.946282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.946282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944744    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.946282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.944782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.946282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.945589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.946282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.945727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.947481 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.946747    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.947481 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.946823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.947481 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.947426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.950027 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.949997    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951483 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.950989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951483 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.951048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951483 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.951185    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951483 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.951425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951483 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.951434    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951809 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.951533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.951869 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.951783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.952581 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.952548    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.952672 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.952629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.952742 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.952721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.953044 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.953025    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.954002 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.953953    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.955277 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.955245    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:08.955704 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:08.955683    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ghk\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-kube-api-access-x7ghk\") pod \"prometheus-k8s-0\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:09.114989 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:09.114946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:11.184685 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:11.184639    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:11.185148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:11.184717    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:11.190531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:11.190502    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:11.198992 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:11.198956    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:12.710227 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:12.710158    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8665cddf46-kq8wl"]
Jul 09 16:52:12.797432 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:12.797402    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:52:13.211164 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.211121    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g" event={"ID":"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479","Type":"ContainerStarted","Data":"719df14009249f9642e43225c321383c34021a8239711e535cf98d308bc2252c"}
Jul 09 16:52:13.211342 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.211200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g" event={"ID":"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479","Type":"ContainerStarted","Data":"8527cb9722b609bce9c5a6c3606611a887ca3471f13eef5e88680dba3c436bc5"}
Jul 09 16:52:13.211342 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.211232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g" event={"ID":"4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479","Type":"ContainerStarted","Data":"c0cd5b50d233dd32445b8627c299f203019f47452af2332732695705d2e85a04"}
Jul 09 16:52:13.213019 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.212984    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bh5ql" event={"ID":"1bdb1bd7-4822-45f9-a873-a819a53b6042","Type":"ContainerStarted","Data":"a7c96ed282ab8fb179d3cdbfc6fb40bff1f1e89faa1eee00f080b09918d5c8dd"}
Jul 09 16:52:13.213178 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.213070    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:52:13.215575 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.215157    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-884hx" event={"ID":"2cff268e-e6cc-4a37-bac8-b79cd442a131","Type":"ContainerStarted","Data":"70c8f60f1f10d9c21f4008c14f535d4b02df0a76fb5eb0360c092211e53f7b6a"}
Jul 09 16:52:13.215575 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.215211    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-884hx" event={"ID":"2cff268e-e6cc-4a37-bac8-b79cd442a131","Type":"ContainerStarted","Data":"802c2e5f3331416b6aebbffa2a54d81dac9429e28b921892593ec40092cdd2ab"}
Jul 09 16:52:13.217437 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.217413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4" event={"ID":"d2683169-3272-4fda-9422-bb4e082a59da","Type":"ContainerStarted","Data":"571dc7403a45690351333161195354764f4e8bcd1079703f303827bff6a50de3"}
Jul 09 16:52:13.219214 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.219189    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac" exitCode=0
Jul 09 16:52:13.219318 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.219270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac"}
Jul 09 16:52:13.222232 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.222208    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"afe2d36016c2461a2b2220c915740f5edf20a68c5d4ee1ecb36564ea61d86789"}
Jul 09 16:52:13.222365 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.222262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"0b7c7b83089e93f1e4efa85d8919cfe53c36c92829ffd88a414b9869bbeff2bc"}
Jul 09 16:52:13.222365 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.222288    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"3812516521edbbd4b411a67e8ba50cd843d64db0f7e88adc9c88266860409083"}
Jul 09 16:52:13.223999 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.223949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fx9l" event={"ID":"77fbfceb-30f2-4e00-9a5f-c083f062c845","Type":"ContainerStarted","Data":"10a2c3485c414622cf3a725c6585144032220fc9e1c49b07b3ee8c314c917f7d"}
Jul 09 16:52:13.225746 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.225719    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343" exitCode=0
Jul 09 16:52:13.225908 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.225819    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"}
Jul 09 16:52:13.225908 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.225893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"4de83f5750f77d0ee4ad077f41008eb75d5657dbccd09d85a1a8d885ad131cc3"}
Jul 09 16:52:13.227185 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.227159    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl" event={"ID":"27e3c04e-029e-44be-bc68-cae624d998cd","Type":"ContainerStarted","Data":"014c13222b0d2810454bb97631570bff3acf8569e52460c8a568402fc84b5ec5"}
Jul 09 16:52:13.229478 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.229418    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ljgb9" event={"ID":"7820052a-8a08-454a-9f39-c861fc961261","Type":"ContainerStarted","Data":"77f9287a0d426d8fd54207df2f811540bfbe4fcae18e7d49fd7902bee353315e"}
Jul 09 16:52:13.233064 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.233014    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-fk57g" podStartSLOduration=4.92881917 podStartE2EDuration="11.232994578s" totalImagesPullingTime="6.304175408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:13.231586206 +0000 UTC m=+73.928758533" watchObservedRunningTime="2026-07-09 16:52:13.232994578 +0000 UTC m=+73.930166905"
Jul 09 16:52:13.282602 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.282492    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bh5ql" podStartSLOduration=67.79847053 podStartE2EDuration="1m14.282471156s" totalImagesPullingTime="6.484000626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:13.281106565 +0000 UTC m=+73.978278893" watchObservedRunningTime="2026-07-09 16:52:13.282471156 +0000 UTC m=+73.979643483"
Jul 09 16:52:13.299963 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.299912    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ljgb9" podStartSLOduration=1.68983168 podStartE2EDuration="19.299894683s" totalImagesPullingTime="17.610063003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:51:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:13.299472096 +0000 UTC m=+73.996644418" watchObservedRunningTime="2026-07-09 16:52:13.299894683 +0000 UTC m=+73.997066988"
Jul 09 16:52:13.341743 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.341689    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-gzpb4" podStartSLOduration=4.932936118 podStartE2EDuration="11.341671191s" totalImagesPullingTime="6.408735073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:13.341171427 +0000 UTC m=+74.038343754" watchObservedRunningTime="2026-07-09 16:52:13.341671191 +0000 UTC m=+74.038843517"
Jul 09 16:52:13.443238 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.443036    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-884hx" podStartSLOduration=67.982689046 podStartE2EDuration="1m14.443017918s" totalImagesPullingTime="6.460328872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:13.44206392 +0000 UTC m=+74.139236247" watchObservedRunningTime="2026-07-09 16:52:13.443017918 +0000 UTC m=+74.140190286"
Jul 09 16:52:13.475056 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:13.474124    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4fx9l" podStartSLOduration=67.89515788 podStartE2EDuration="1m14.474099923s" totalImagesPullingTime="6.578942043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:13.473037956 +0000 UTC m=+74.170210483" watchObservedRunningTime="2026-07-09 16:52:13.474099923 +0000 UTC m=+74.171272252"
Jul 09 16:52:15.243684 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.243640    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"8ca19eed123da66486ca2dde8e6fa6ca77ee0f7439ac2738692df3b017fedf48"}
Jul 09 16:52:15.244193 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.243737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"fbcdf2a7f13f7f3ed0aa75183b4129dad3c59bba274716cbf92d87da3e8061e7"}
Jul 09 16:52:15.244193 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.243781    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" event={"ID":"87a7410d-336a-4e97-9507-f7432751a933","Type":"ContainerStarted","Data":"8c84eb224f6126f20c1e9ca7507b1f85e2d124f23d4fe43c3b3131727b5ba437"}
Jul 09 16:52:15.244193 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.243868    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:15.247899 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.247864    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl" event={"ID":"27e3c04e-029e-44be-bc68-cae624d998cd","Type":"ContainerStarted","Data":"478b3d7dfd923df5f5bb563557198a1170e79afa5054b5e00445a7f9bed52911"}
Jul 09 16:52:15.271710 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.268330    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml" podStartSLOduration=2.627274452 podStartE2EDuration="11.268306952s" totalImagesPullingTime="8.6410325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:15.266777778 +0000 UTC m=+75.963950121" watchObservedRunningTime="2026-07-09 16:52:15.268306952 +0000 UTC m=+75.965479279"
Jul 09 16:52:15.286543 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:15.285509    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl" podStartSLOduration=7.79513421 podStartE2EDuration="9.285487225s" totalImagesPullingTime="1.490353015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:15.283539778 +0000 UTC m=+75.980712106" watchObservedRunningTime="2026-07-09 16:52:15.285487225 +0000 UTC m=+75.982659554"
Jul 09 16:52:16.257938 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:16.257888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79"}
Jul 09 16:52:17.268013 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.267971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e"}
Jul 09 16:52:17.268647 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.268073    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537"}
Jul 09 16:52:17.268647 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.268115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717"}
Jul 09 16:52:17.268647 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.268156    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c"}
Jul 09 16:52:17.273527 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.273472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"}
Jul 09 16:52:17.273639 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.273609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"}
Jul 09 16:52:17.273698 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.273683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"}
Jul 09 16:52:17.273768 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:17.273754    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"}
Jul 09 16:52:18.280876 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:18.280838    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerStarted","Data":"22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed"}
Jul 09 16:52:18.286760 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:18.286724    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"}
Jul 09 16:52:18.286911 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:18.286820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerStarted","Data":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"}
Jul 09 16:52:18.312207 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:18.312136    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.278887138 podStartE2EDuration="15.312112153s" totalImagesPullingTime="9.002511424s" totalInitContainerRuntime="30.713591ms" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:18.309584433 +0000 UTC m=+79.006756760" watchObservedRunningTime="2026-07-09 16:52:18.312112153 +0000 UTC m=+79.009284480"
Jul 09 16:52:18.346878 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:18.342874    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.662023668 podStartE2EDuration="10.342851155s" totalImagesPullingTime="3.644947817s" totalInitContainerRuntime="35.87967ms" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:18.340234499 +0000 UTC m=+79.037406841" watchObservedRunningTime="2026-07-09 16:52:18.342851155 +0000 UTC m=+79.040023484"
Jul 09 16:52:19.116167 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:19.116125    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:52:21.264609 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:21.264476    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d56995d65-8sxml"
Jul 09 16:52:27.229892 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:27.229853    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:27.230319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:27.229922    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:28.267940 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:28.267905    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f7594b87-wp8zq"]
Jul 09 16:52:44.238153 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:44.238117    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bh5ql"
Jul 09 16:52:47.235401 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:47.235364    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:47.239866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:47.239843    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8665cddf46-kq8wl"
Jul 09 16:52:53.289259 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.289208    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57f7594b87-wp8zq" podUID="da48f9a4-3950-42c9-9228-017301d59839" containerName="console" containerID="cri-o://5ff753344e0c2a4244045105dec2a670da17584db8a25ee09d1af0aa2454885c" gracePeriod=15
Jul 09 16:52:53.438578 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.438547    2567 generic.go:356] "Generic (PLEG): container finished" podID="da48f9a4-3950-42c9-9228-017301d59839" containerID="5ff753344e0c2a4244045105dec2a670da17584db8a25ee09d1af0aa2454885c" exitCode=0
Jul 09 16:52:53.438746 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.438622    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f7594b87-wp8zq" event={"ID":"da48f9a4-3950-42c9-9228-017301d59839","Type":"ContainerDied","Data":"5ff753344e0c2a4244045105dec2a670da17584db8a25ee09d1af0aa2454885c"}
Jul 09 16:52:53.534649 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.534625    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:53.648121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648089    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-service-ca\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648133    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-console-config\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648156    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-serving-cert\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648180    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b4bn\" (UniqueName: \"kubernetes.io/projected/da48f9a4-3950-42c9-9228-017301d59839-kube-api-access-4b4bn\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648220    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-oauth-config\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648247    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-oauth-serving-cert\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648289    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-trusted-ca-bundle\") pod \"da48f9a4-3950-42c9-9228-017301d59839\" (UID: \"da48f9a4-3950-42c9-9228-017301d59839\") "
Jul 09 16:52:53.648713 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648679    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-console-config" (OuterVolumeSpecName: "console-config") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:52:53.648818 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648705    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:52:53.648818 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648746    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-service-ca" (OuterVolumeSpecName: "service-ca") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:52:53.648818 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.648740    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:52:53.650842 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.650814    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:52:53.650941 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.650811    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da48f9a4-3950-42c9-9228-017301d59839-kube-api-access-4b4bn" (OuterVolumeSpecName: "kube-api-access-4b4bn") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "kube-api-access-4b4bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:52:53.651000 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.650928    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da48f9a4-3950-42c9-9228-017301d59839" (UID: "da48f9a4-3950-42c9-9228-017301d59839"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:52:53.749371 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749320    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4bn\" (UniqueName: \"kubernetes.io/projected/da48f9a4-3950-42c9-9228-017301d59839-kube-api-access-4b4bn\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:53.749371 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749366    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:53.749371 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749377    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:53.749371 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749385    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:53.749371 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749394    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:53.749686 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749402    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da48f9a4-3950-42c9-9228-017301d59839-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:53.749686 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:53.749410    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da48f9a4-3950-42c9-9228-017301d59839-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:52:54.443580 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:54.443549    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f7594b87-wp8zq"
Jul 09 16:52:54.444090 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:54.443578    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f7594b87-wp8zq" event={"ID":"da48f9a4-3950-42c9-9228-017301d59839","Type":"ContainerDied","Data":"f71aad705046a2f54d306deae73c6b8b932eb447e5a3ca7ed885d1b6c1525fd9"}
Jul 09 16:52:54.444090 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:54.443640    2567 scope.go:117] "RemoveContainer" containerID="5ff753344e0c2a4244045105dec2a670da17584db8a25ee09d1af0aa2454885c"
Jul 09 16:52:54.460938 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:54.460913    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f7594b87-wp8zq"]
Jul 09 16:52:54.464564 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:54.464539    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57f7594b87-wp8zq"]
Jul 09 16:52:55.871238 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:52:55.871206    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da48f9a4-3950-42c9-9228-017301d59839" path="/var/lib/kubelet/pods/da48f9a4-3950-42c9-9228-017301d59839/volumes"
Jul 09 16:53:09.115809 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:09.115765    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:09.136207 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:09.136182    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:09.517981 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:09.517902    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:22.889706 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:22.889671    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 16:53:23.553910 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:23.553834    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="alertmanager" containerID="cri-o://6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79" gracePeriod=120
Jul 09 16:53:23.554120 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:23.553907    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-web" containerID="cri-o://8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717" gracePeriod=120
Jul 09 16:53:23.554120 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:23.553873    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-metric" containerID="cri-o://1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e" gracePeriod=120
Jul 09 16:53:23.554120 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:23.553933    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="prom-label-proxy" containerID="cri-o://22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed" gracePeriod=120
Jul 09 16:53:23.554120 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:23.553942    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="config-reloader" containerID="cri-o://bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c" gracePeriod=120
Jul 09 16:53:23.554120 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:23.553924    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy" containerID="cri-o://4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537" gracePeriod=120
Jul 09 16:53:24.560840 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560785    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed" exitCode=0
Jul 09 16:53:24.560840 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560836    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e" exitCode=0
Jul 09 16:53:24.560840 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560847    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537" exitCode=0
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560854    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c" exitCode=0
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560861    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79" exitCode=0
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560831    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed"}
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560943    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e"}
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560973    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537"}
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.560996    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c"}
Jul 09 16:53:24.561298 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.561021    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79"}
Jul 09 16:53:24.789228 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.789206    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:53:24.823080 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823011    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcsdk\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-kube-api-access-qcsdk\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823080 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823049    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823086    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823131    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-web\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823166    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-out\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823224    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-trusted-ca-bundle\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823261 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823251    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-metrics-client-ca\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823274    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-web-config\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823321    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823351    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-tls-assets\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823375    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-main-db\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823398    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-cluster-tls-config\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.823774 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823726    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:24.824013 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.823992    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-volume\") pod \"aede80d7-03eb-43cd-ac80-cf585225cc8e\" (UID: \"aede80d7-03eb-43cd-ac80-cf585225cc8e\") "
Jul 09 16:53:24.824263 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.824161    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:24.824334 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.824310    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.824392 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.824342    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede80d7-03eb-43cd-ac80-cf585225cc8e-metrics-client-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.826088 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.826057    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:53:24.829833 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.829771    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.830184 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.830153    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-out" (OuterVolumeSpecName: "config-out") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:53:24.830411 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.830316    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.830411 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.830365    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:53:24.830538 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.830506    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-kube-api-access-qcsdk" (OuterVolumeSpecName: "kube-api-access-qcsdk") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "kube-api-access-qcsdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:53:24.830727 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.830703    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.830893 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.830872    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.831957 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.831935    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-volume" (OuterVolumeSpecName: "config-volume") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.834480 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.834452    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.841767 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.841728    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-web-config" (OuterVolumeSpecName: "web-config") pod "aede80d7-03eb-43cd-ac80-cf585225cc8e" (UID: "aede80d7-03eb-43cd-ac80-cf585225cc8e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:24.925331 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925309    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-web-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925331 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925332    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925342    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-tls-assets\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925351    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-alertmanager-main-db\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925360    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-cluster-tls-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925369    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925377    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsdk\" (UniqueName: \"kubernetes.io/projected/aede80d7-03eb-43cd-ac80-cf585225cc8e-kube-api-access-qcsdk\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925385    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-main-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925394    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925402    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aede80d7-03eb-43cd-ac80-cf585225cc8e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:24.925469 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:24.925410    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aede80d7-03eb-43cd-ac80-cf585225cc8e-config-out\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:25.567899 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.567865    2567 generic.go:356] "Generic (PLEG): container finished" podID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerID="8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717" exitCode=0
Jul 09 16:53:25.568267 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.567945    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 16:53:25.568267 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.567965    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717"}
Jul 09 16:53:25.568267 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.568045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aede80d7-03eb-43cd-ac80-cf585225cc8e","Type":"ContainerDied","Data":"b9c567f8ec8815d21b3c211479fc6ecbb31ea943a350f0968eabe1391e11f446"}
Jul 09 16:53:25.568267 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.568082    2567 scope.go:117] "RemoveContainer" containerID="22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed"
Jul 09 16:53:25.576790 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.576773    2567 scope.go:117] "RemoveContainer" containerID="1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e"
Jul 09 16:53:25.583971 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.583955    2567 scope.go:117] "RemoveContainer" containerID="4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537"
Jul 09 16:53:25.590618 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.590602    2567 scope.go:117] "RemoveContainer" containerID="8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717"
Jul 09 16:53:25.597515 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.597493    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 16:53:25.598424 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.598407    2567 scope.go:117] "RemoveContainer" containerID="bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c"
Jul 09 16:53:25.601315 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.601295    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 16:53:25.605659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.605640    2567 scope.go:117] "RemoveContainer" containerID="6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79"
Jul 09 16:53:25.612466 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.612445    2567 scope.go:117] "RemoveContainer" containerID="e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac"
Jul 09 16:53:25.620639 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.620624    2567 scope.go:117] "RemoveContainer" containerID="22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed"
Jul 09 16:53:25.620920 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.620900    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed\": container with ID starting with 22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed not found: ID does not exist" containerID="22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed"
Jul 09 16:53:25.620976 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.620930    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed"} err="failed to get container status \"22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed\": rpc error: code = NotFound desc = could not find container \"22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed\": container with ID starting with 22ed3292ee7729610cde5fb16f7a42b2118924b19d385f9d4e971098ca044bed not found: ID does not exist"
Jul 09 16:53:25.620976 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.620961    2567 scope.go:117] "RemoveContainer" containerID="1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e"
Jul 09 16:53:25.621251 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.621222    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e\": container with ID starting with 1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e not found: ID does not exist" containerID="1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e"
Jul 09 16:53:25.621293 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.621258    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e"} err="failed to get container status \"1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e\": rpc error: code = NotFound desc = could not find container \"1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e\": container with ID starting with 1d545e123921b141a036c3234f708a6973df6179647dec650f560972c152496e not found: ID does not exist"
Jul 09 16:53:25.621293 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.621274    2567 scope.go:117] "RemoveContainer" containerID="4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537"
Jul 09 16:53:25.621532 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.621514    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537\": container with ID starting with 4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537 not found: ID does not exist" containerID="4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537"
Jul 09 16:53:25.621572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.621539    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537"} err="failed to get container status \"4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537\": rpc error: code = NotFound desc = could not find container \"4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537\": container with ID starting with 4ce6294424c84188a4b09aaceb36afe5d477819302725c70320458b89ddde537 not found: ID does not exist"
Jul 09 16:53:25.621572 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.621556    2567 scope.go:117] "RemoveContainer" containerID="8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717"
Jul 09 16:53:25.621780 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.621761    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717\": container with ID starting with 8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717 not found: ID does not exist" containerID="8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717"
Jul 09 16:53:25.621866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.621788    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717"} err="failed to get container status \"8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717\": rpc error: code = NotFound desc = could not find container \"8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717\": container with ID starting with 8448e09d18296ed69b818b90d59830d4cb69cdbdb3473df0c2efa58db73c6717 not found: ID does not exist"
Jul 09 16:53:25.621866 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.621825    2567 scope.go:117] "RemoveContainer" containerID="bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c"
Jul 09 16:53:25.622091 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.622073    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c\": container with ID starting with bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c not found: ID does not exist" containerID="bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c"
Jul 09 16:53:25.622152 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.622104    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c"} err="failed to get container status \"bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c\": rpc error: code = NotFound desc = could not find container \"bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c\": container with ID starting with bd299e448a52e2f31bf0374a4584492f2155350cb9d592b2a47f06e948a7319c not found: ID does not exist"
Jul 09 16:53:25.622152 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.622118    2567 scope.go:117] "RemoveContainer" containerID="6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79"
Jul 09 16:53:25.622338 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.622320    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79\": container with ID starting with 6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79 not found: ID does not exist" containerID="6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79"
Jul 09 16:53:25.622381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.622343    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79"} err="failed to get container status \"6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79\": rpc error: code = NotFound desc = could not find container \"6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79\": container with ID starting with 6d9076c4e22babd86e6aaa9623a0e9917c0003b4286e76ffaccb2fbebd357b79 not found: ID does not exist"
Jul 09 16:53:25.622381 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.622362    2567 scope.go:117] "RemoveContainer" containerID="e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac"
Jul 09 16:53:25.622565 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:25.622552    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac\": container with ID starting with e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac not found: ID does not exist" containerID="e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac"
Jul 09 16:53:25.622607 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.622567    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac"} err="failed to get container status \"e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac\": rpc error: code = NotFound desc = could not find container \"e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac\": container with ID starting with e8eec5b7bca42fb9851fff7a34c6153dd9654399cc15b5dfa8be29eeb0fd61ac not found: ID does not exist"
Jul 09 16:53:25.875566 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:25.875529    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" path="/var/lib/kubelet/pods/aede80d7-03eb-43cd-ac80-cf585225cc8e/volumes"
Jul 09 16:53:27.113257 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113216    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:53:27.113659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113504    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="prometheus" containerID="cri-o://0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790" gracePeriod=600
Jul 09 16:53:27.113659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113540    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy" containerID="cri-o://0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143" gracePeriod=600
Jul 09 16:53:27.113659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113575    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="thanos-sidecar" containerID="cri-o://780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1" gracePeriod=600
Jul 09 16:53:27.113659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113593    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="config-reloader" containerID="cri-o://1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e" gracePeriod=600
Jul 09 16:53:27.113659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113602    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a" gracePeriod=600
Jul 09 16:53:27.113962 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.113598    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-web" containerID="cri-o://5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab" gracePeriod=600
Jul 09 16:53:27.359026 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.359003    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.444660 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444580    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-metrics-client-ca\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444660 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444623    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-tls\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444660 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444644    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444926 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444681    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444926 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444819    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-web-config\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444926 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444853    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-0\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444926 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444882    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-serving-certs-ca-bundle\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.444926 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444906    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-db\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444956    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-trusted-ca-bundle\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.444990    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-thanos-prometheus-http-client-file\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7ghk\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-kube-api-access-x7ghk\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445054    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-grpc-tls\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445095    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-kube-rbac-proxy\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445148    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-metrics-client-certs\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445152    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.445182 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445173    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-2\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445202    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-kubelet-serving-ca-bundle\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445229    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config-out\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445255    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-tls-assets\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445354    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-1\") pod \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\" (UID: \"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5\") "
Jul 09 16:53:27.445777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445569    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.445777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445719    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.446109 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.445904    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.446642 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.446615    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-metrics-client-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.447258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.447166    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.447258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.447252    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.448035 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.447525    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.448343 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.448273    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:53:27.448343 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.448305    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.448780 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.448700    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config" (OuterVolumeSpecName: "config") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.449283 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.449154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:53:27.449283 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.449273    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.449653 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.449626    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.449860 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.449789    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.449947 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.449901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-kube-api-access-x7ghk" (OuterVolumeSpecName: "kube-api-access-x7ghk") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "kube-api-access-x7ghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:53:27.450004 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.449970    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.450485 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.450455    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.450727 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.450699    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.450905 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.450750    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config-out" (OuterVolumeSpecName: "config-out") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:53:27.451071 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.451054    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:53:27.458254 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.458236    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-web-config" (OuterVolumeSpecName: "web-config") pod "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" (UID: "17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:53:27.547748 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547718    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547748 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547748    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ghk\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-kube-api-access-x7ghk\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547748 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547757    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-grpc-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547775    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547784    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-kube-rbac-proxy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547808    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-metrics-client-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547818    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547826    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547834    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-config-out\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547842    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-tls-assets\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547849    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547857    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547867    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547877    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547885    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-web-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547894    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547903    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.547935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.547912    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5-prometheus-k8s-db\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:53:27.579957 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579931    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a" exitCode=0
Jul 09 16:53:27.579957 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579956    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143" exitCode=0
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579966    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab" exitCode=0
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579974    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1" exitCode=0
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579981    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e" exitCode=0
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579989    2567 generic.go:356] "Generic (PLEG): container finished" podID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790" exitCode=0
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.579994    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"}
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580029    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.580074 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580061    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"}
Jul 09 16:53:27.580268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580089    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"}
Jul 09 16:53:27.580268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580113    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"}
Jul 09 16:53:27.580268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580135    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.580268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580137    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"}
Jul 09 16:53:27.580268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"}
Jul 09 16:53:27.580268 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.580264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5","Type":"ContainerDied","Data":"4de83f5750f77d0ee4ad077f41008eb75d5657dbccd09d85a1a8d885ad131cc3"}
Jul 09 16:53:27.588942 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.588912    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.596905 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.596890    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.603475 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.603453    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:53:27.604005 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.603991    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.607958 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.607939    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:53:27.610930 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.610915    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.617291 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.617275    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.624639 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.624623    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.631367 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.631351    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.631612 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.631597    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.631656 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.631621    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"} err="failed to get container status \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist"
Jul 09 16:53:27.631656 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.631641    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.631999 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.631939    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.632590 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.632091    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"} err="failed to get container status \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist"
Jul 09 16:53:27.632676 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.632598    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.632966 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.632944    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.633077 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.632970    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"} err="failed to get container status \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist"
Jul 09 16:53:27.633077 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.632987    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.633244 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.633225    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.633304 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633251    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"} err="failed to get container status \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist"
Jul 09 16:53:27.633304 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633273    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.633504 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.633490    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.633560 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633512    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:53:27.633560 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633512    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"} err="failed to get container status \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist"
Jul 09 16:53:27.633560 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633531    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.633749 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.633731    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.633816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633753    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-web"
Jul 09 16:53:27.633816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633764    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-web"
Jul 09 16:53:27.633816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633757    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"} err="failed to get container status \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist"
Jul 09 16:53:27.633816 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633783    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633774    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-metric"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633848    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-metric"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633859    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da48f9a4-3950-42c9-9228-017301d59839" containerName="console"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633865    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="da48f9a4-3950-42c9-9228-017301d59839" containerName="console"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633872    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633877    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633886    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="config-reloader"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633895    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="config-reloader"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633907    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633912    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633918    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="prom-label-proxy"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633924    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="prom-label-proxy"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633930    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="init-config-reloader"
Jul 09 16:53:27.633935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633936    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="init-config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633943    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="alertmanager"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633948    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="alertmanager"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633954    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633959    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633972    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="prometheus"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633978    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="prometheus"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633985    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="init-config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633990    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="init-config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.633995    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-thanos"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634000    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-thanos"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634007    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-web"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634012    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-web"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634018    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="thanos-sidecar"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634022    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="thanos-sidecar"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:53:27.634043    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634060    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"} err="failed to get container status \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634077    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634069    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="prometheus"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634126    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="alertmanager"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634134    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-web"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634140    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634147    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634156    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="config-reloader"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634165    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="thanos-sidecar"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634171    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy-metric"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634179    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="kube-rbac-proxy"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634186    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-web"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634193    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" containerName="kube-rbac-proxy-thanos"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634201    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="da48f9a4-3950-42c9-9228-017301d59839" containerName="console"
Jul 09 16:53:27.634395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634209    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede80d7-03eb-43cd-ac80-cf585225cc8e" containerName="prom-label-proxy"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634346    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"} err="failed to get container status \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634366    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634573    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"} err="failed to get container status \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634592    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634814    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"} err="failed to get container status \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.634831    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635062    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"} err="failed to get container status \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635083    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635324    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"} err="failed to get container status \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist"
Jul 09 16:53:27.635451 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635339    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.635860 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635536    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"} err="failed to get container status \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist"
Jul 09 16:53:27.635860 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635553    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.635860 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635760    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"} err="failed to get container status \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist"
Jul 09 16:53:27.635860 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.635808    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.636041 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636023    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"} err="failed to get container status \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist"
Jul 09 16:53:27.636097 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636042    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.636238 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636219    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"} err="failed to get container status \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist"
Jul 09 16:53:27.636306 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636239    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.636472 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636457    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"} err="failed to get container status \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist"
Jul 09 16:53:27.636538 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636475    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.636695 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636675    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"} err="failed to get container status \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist"
Jul 09 16:53:27.636740 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636695    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.636945 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636925    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"} err="failed to get container status \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist"
Jul 09 16:53:27.636945 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.636945    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.637174 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637156    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"} err="failed to get container status \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist"
Jul 09 16:53:27.637222 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637175    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.637222 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637179    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.637597 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637406    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"} err="failed to get container status \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist"
Jul 09 16:53:27.637597 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637436    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.637825 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637776    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"} err="failed to get container status \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist"
Jul 09 16:53:27.637939 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.637825    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.638135 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638106    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"} err="failed to get container status \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist"
Jul 09 16:53:27.638135 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638135    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.638387 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638359    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"} err="failed to get container status \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist"
Jul 09 16:53:27.638387 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638379    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.638611 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638590    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"} err="failed to get container status \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist"
Jul 09 16:53:27.638659 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638615    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.638889 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638866    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"} err="failed to get container status \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist"
Jul 09 16:53:27.638889 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.638890    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.639138 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639107    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"} err="failed to get container status \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist"
Jul 09 16:53:27.639138 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639138    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.639432 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639408    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"} err="failed to get container status \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist"
Jul 09 16:53:27.639500 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639435    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.639709 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639688    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"} err="failed to get container status \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist"
Jul 09 16:53:27.639812 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639712    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.639956 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.639937    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 16:53:27.640085 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640068    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 16:53:27.640170 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640155    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 16:53:27.640345 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640327    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"} err="failed to get container status \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist"
Jul 09 16:53:27.640400 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640350    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.640400 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 16:53:27.640531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640428    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 16:53:27.640531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640484    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 16:53:27.640633 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640577    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 16:53:27.640765 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640740    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 16:53:27.640852 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640773    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"} err="failed to get container status \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist"
Jul 09 16:53:27.640852 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.640819    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.641073 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641053    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"} err="failed to get container status \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist"
Jul 09 16:53:27.641073 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641072    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.641251 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641238    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3arht6r5pkgqc\""
Jul 09 16:53:27.641318 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641297    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"} err="failed to get container status \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist"
Jul 09 16:53:27.641367 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641320    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 16:53:27.641367 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641241    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 16:53:27.641455 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641322    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.641497 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641482    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 16:53:27.641534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641434    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 16:53:27.641708 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641680    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rtrff\""
Jul 09 16:53:27.641852 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641734    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"} err="failed to get container status \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist"
Jul 09 16:53:27.641852 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.641755    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.642093 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642068    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"} err="failed to get container status \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist"
Jul 09 16:53:27.642178 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642093    2567 scope.go:117] "RemoveContainer" containerID="4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"
Jul 09 16:53:27.642366 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642342    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a"} err="failed to get container status \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": rpc error: code = NotFound desc = could not find container \"4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a\": container with ID starting with 4f1751542eb5700c23930f51ec76da6aeb9968a118690cdb06f38a90f997465a not found: ID does not exist"
Jul 09 16:53:27.642445 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642368    2567 scope.go:117] "RemoveContainer" containerID="0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"
Jul 09 16:53:27.642751 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642639    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143"} err="failed to get container status \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": rpc error: code = NotFound desc = could not find container \"0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143\": container with ID starting with 0ff10d41f6e6ac47569ff6afbcf7c0984af6050db1ef0f2f3477f38e27c8d143 not found: ID does not exist"
Jul 09 16:53:27.642751 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642663    2567 scope.go:117] "RemoveContainer" containerID="5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"
Jul 09 16:53:27.642966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642942    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab"} err="failed to get container status \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": rpc error: code = NotFound desc = could not find container \"5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab\": container with ID starting with 5d0db99f66fab44a29314b1c7bc5615b0629d57930977de72aada9b6eaec2cab not found: ID does not exist"
Jul 09 16:53:27.642966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.642967    2567 scope.go:117] "RemoveContainer" containerID="780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"
Jul 09 16:53:27.643327 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643222    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1"} err="failed to get container status \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": rpc error: code = NotFound desc = could not find container \"780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1\": container with ID starting with 780b98687a2a165b1d253ac9dea129981d7897dae5ac2dc1305fe7b8d38affd1 not found: ID does not exist"
Jul 09 16:53:27.643327 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643245    2567 scope.go:117] "RemoveContainer" containerID="1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"
Jul 09 16:53:27.643327 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643296    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 16:53:27.643500 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643477    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e"} err="failed to get container status \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": rpc error: code = NotFound desc = could not find container \"1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e\": container with ID starting with 1bd04aae8dcb833e40e8bdfc736ebd014dafbca065aba4e241dbd1bcd68f100e not found: ID does not exist"
Jul 09 16:53:27.643546 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643502    2567 scope.go:117] "RemoveContainer" containerID="0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"
Jul 09 16:53:27.643758 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643733    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790"} err="failed to get container status \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": rpc error: code = NotFound desc = could not find container \"0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790\": container with ID starting with 0a2f86fdadc1b335c285c7b2893c422f34b127e1432d3ff932b24ea1d7a19790 not found: ID does not exist"
Jul 09 16:53:27.643845 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.643761    2567 scope.go:117] "RemoveContainer" containerID="74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"
Jul 09 16:53:27.644037 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.644013    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343"} err="failed to get container status \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": rpc error: code = NotFound desc = could not find container \"74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343\": container with ID starting with 74a5c03e9f468e9556c6248c139e2cb42c9915487f848bab97231a2281480343 not found: ID does not exist"
Jul 09 16:53:27.646112 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.646091    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 16:53:27.655174 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.655144    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:53:27.748694 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/187f2790-8a81-47f8-80a0-1cf05558c9e0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748694 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pvg\" (UniqueName: \"kubernetes.io/projected/187f2790-8a81-47f8-80a0-1cf05558c9e0-kube-api-access-n4pvg\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748694 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748691    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748871 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748871 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748871 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748871 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748827    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.748871 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748852    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748877    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749010 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.748989    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749033    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-config\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/187f2790-8a81-47f8-80a0-1cf05558c9e0-config-out\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749090    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-web-config\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749148 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.749314 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.749148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850107 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850107    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850150    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850172    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850200 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850328    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-config\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850410 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850408    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/187f2790-8a81-47f8-80a0-1cf05558c9e0-config-out\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850429    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-web-config\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/187f2790-8a81-47f8-80a0-1cf05558c9e0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.850772 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.850562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pvg\" (UniqueName: \"kubernetes.io/projected/187f2790-8a81-47f8-80a0-1cf05558c9e0-kube-api-access-n4pvg\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.851571    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.851781    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.852217    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.852448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853121 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.852964    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853508 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.853480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853843 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.853822    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.853843 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.853826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-config\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.854281 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.854021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.854281 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.853840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.854281 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.854103    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/187f2790-8a81-47f8-80a0-1cf05558c9e0-config-out\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.854646 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.854586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.854863 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.854839    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.856088 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.856066    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.856412 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.856392    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/187f2790-8a81-47f8-80a0-1cf05558c9e0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.856522 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.856507    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/187f2790-8a81-47f8-80a0-1cf05558c9e0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.856664 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.856648    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.856729 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.856712    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/187f2790-8a81-47f8-80a0-1cf05558c9e0-web-config\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.858338 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.858321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pvg\" (UniqueName: \"kubernetes.io/projected/187f2790-8a81-47f8-80a0-1cf05558c9e0-kube-api-access-n4pvg\") pod \"prometheus-k8s-0\" (UID: \"187f2790-8a81-47f8-80a0-1cf05558c9e0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:27.875166 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.875139    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5" path="/var/lib/kubelet/pods/17a9ba2e-9d50-4a7d-97ef-1bc88a9143f5/volumes"
Jul 09 16:53:27.948318 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:27.948287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:53:28.084380 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:28.084352    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 16:53:28.087395 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:53:28.087373    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187f2790_8a81_47f8_80a0_1cf05558c9e0.slice/crio-ab484e54e99f0096fc2a6937bd7e10013c684ccc3f17e312a35f9fdd28c74d65 WatchSource:0}: Error finding container ab484e54e99f0096fc2a6937bd7e10013c684ccc3f17e312a35f9fdd28c74d65: Status 404 returned error can't find the container with id ab484e54e99f0096fc2a6937bd7e10013c684ccc3f17e312a35f9fdd28c74d65
Jul 09 16:53:28.586296 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:28.586262    2567 generic.go:356] "Generic (PLEG): container finished" podID="187f2790-8a81-47f8-80a0-1cf05558c9e0" containerID="191570ab1ef9ee977c30e1e8d42daf1d6a02331a379ddb00f4a34ec6912a0cdd" exitCode=0
Jul 09 16:53:28.586661 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:28.586340    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerDied","Data":"191570ab1ef9ee977c30e1e8d42daf1d6a02331a379ddb00f4a34ec6912a0cdd"}
Jul 09 16:53:28.586661 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:28.586407    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"ab484e54e99f0096fc2a6937bd7e10013c684ccc3f17e312a35f9fdd28c74d65"}
Jul 09 16:53:29.593871 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.593833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"017d9df431f79fe67a7c08aa7c7b7ed30a75ecb60f20986e254901f3ee28cbb1"}
Jul 09 16:53:29.594240 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.593904    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"2ae0f8a2f4a23975344b8da0bfafaa4a6216f54e0c0362d762ba7ac8083e746e"}
Jul 09 16:53:29.594240 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.593931    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"27d607ede7c933b7069b440c1fd10f875ee058ebf333800067b18c0542b82c07"}
Jul 09 16:53:29.594240 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.593963    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"039b1d39b7d394bc7bf70860d6685f0b306b6343914f4bbbfb0cb4a7465658c5"}
Jul 09 16:53:29.594240 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.593987    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"35d6ed2511208cca1aae67be837b8d30fe4445fbd9b10487d7adf9fbab0fa834"}
Jul 09 16:53:29.594240 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.594010    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"187f2790-8a81-47f8-80a0-1cf05558c9e0","Type":"ContainerStarted","Data":"07988ea8cb45301473d0bbab6edb50c1fa00be8d569f2d50123d4d7c7c4e868b"}
Jul 09 16:53:29.630552 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:29.630502    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.609940886 podStartE2EDuration="2.630482695s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.541809ms" isStatefulPod=false podCreationTimestamp="2026-07-09 16:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:53:29.627698128 +0000 UTC m=+150.324870456" watchObservedRunningTime="2026-07-09 16:53:29.630482695 +0000 UTC m=+150.327655021"
Jul 09 16:53:32.948734 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:53:32.948697    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:54:27.949258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:54:27.949225    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:54:27.964706 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:54:27.964681    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:54:28.833668 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:54:28.833644    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 16:55:19.987900 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:19.987825    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cvtst"]
Jul 09 16:55:19.989926 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:19.989903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:19.992627 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:19.992601    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 16:55:19.992728 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:19.992696    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zsc5r\""
Jul 09 16:55:19.993777 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:19.993759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 16:55:20.002116 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.002095    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cvtst"]
Jul 09 16:55:20.150578 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.150549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjwk\" (UniqueName: \"kubernetes.io/projected/0178a69f-5ae2-48d3-9555-66f88369121b-kube-api-access-wgjwk\") pod \"cert-manager-55b4d954d5-cvtst\" (UID: \"0178a69f-5ae2-48d3-9555-66f88369121b\") " pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.150719 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.150601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0178a69f-5ae2-48d3-9555-66f88369121b-bound-sa-token\") pod \"cert-manager-55b4d954d5-cvtst\" (UID: \"0178a69f-5ae2-48d3-9555-66f88369121b\") " pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.251818 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.251742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjwk\" (UniqueName: \"kubernetes.io/projected/0178a69f-5ae2-48d3-9555-66f88369121b-kube-api-access-wgjwk\") pod \"cert-manager-55b4d954d5-cvtst\" (UID: \"0178a69f-5ae2-48d3-9555-66f88369121b\") " pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.251818 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.251785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0178a69f-5ae2-48d3-9555-66f88369121b-bound-sa-token\") pod \"cert-manager-55b4d954d5-cvtst\" (UID: \"0178a69f-5ae2-48d3-9555-66f88369121b\") " pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.260387 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.260357    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0178a69f-5ae2-48d3-9555-66f88369121b-bound-sa-token\") pod \"cert-manager-55b4d954d5-cvtst\" (UID: \"0178a69f-5ae2-48d3-9555-66f88369121b\") " pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.260519 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.260505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjwk\" (UniqueName: \"kubernetes.io/projected/0178a69f-5ae2-48d3-9555-66f88369121b-kube-api-access-wgjwk\") pod \"cert-manager-55b4d954d5-cvtst\" (UID: \"0178a69f-5ae2-48d3-9555-66f88369121b\") " pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.306580 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.306557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cvtst"
Jul 09 16:55:20.431679 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.431651    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cvtst"]
Jul 09 16:55:20.434523 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:55:20.434492    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0178a69f_5ae2_48d3_9555_66f88369121b.slice/crio-9e0fa81bf4a4cbfff379112691a81cd197197636db8e7b2cbdab862415eb6201 WatchSource:0}: Error finding container 9e0fa81bf4a4cbfff379112691a81cd197197636db8e7b2cbdab862415eb6201: Status 404 returned error can't find the container with id 9e0fa81bf4a4cbfff379112691a81cd197197636db8e7b2cbdab862415eb6201
Jul 09 16:55:20.990204 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:20.990172    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cvtst" event={"ID":"0178a69f-5ae2-48d3-9555-66f88369121b","Type":"ContainerStarted","Data":"9e0fa81bf4a4cbfff379112691a81cd197197636db8e7b2cbdab862415eb6201"}
Jul 09 16:55:25.008108 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:25.008056    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cvtst" event={"ID":"0178a69f-5ae2-48d3-9555-66f88369121b","Type":"ContainerStarted","Data":"074d0fba6b455f313d22c9b3817c19f82c41ca6015b74845399d851f8d2dadc9"}
Jul 09 16:55:25.034711 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:25.034666    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-cvtst" podStartSLOduration=2.478270567 podStartE2EDuration="6.034650552s" totalImagesPullingTime="3.556379985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:55:25.033880605 +0000 UTC m=+265.731052933" watchObservedRunningTime="2026-07-09 16:55:25.034650552 +0000 UTC m=+265.731822878"
Jul 09 16:55:49.449209 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.449171    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-db6444945-knhg9"]
Jul 09 16:55:49.453652 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.453635    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.457511 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.457489    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:55:49.457511 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.457504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 16:55:49.457692 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.457486    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 16:55:49.457779 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.457758    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 16:55:49.457779 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.457764    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-clfzh\""
Jul 09 16:55:49.457949 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.457878    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 16:55:49.467842 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.467822    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-db6444945-knhg9"]
Jul 09 16:55:49.475767 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.475741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b1e7f49-1803-44d3-977b-5bcae4fe831b-metrics-cert\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.475901 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.475789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1e7f49-1803-44d3-977b-5bcae4fe831b-cert\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.475901 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.475870    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sc9d\" (UniqueName: \"kubernetes.io/projected/0b1e7f49-1803-44d3-977b-5bcae4fe831b-kube-api-access-5sc9d\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.475992 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.475956    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b1e7f49-1803-44d3-977b-5bcae4fe831b-manager-config\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.576783 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.576750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b1e7f49-1803-44d3-977b-5bcae4fe831b-manager-config\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.576966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.576827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b1e7f49-1803-44d3-977b-5bcae4fe831b-metrics-cert\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.576966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.576862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1e7f49-1803-44d3-977b-5bcae4fe831b-cert\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.576966 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.576902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc9d\" (UniqueName: \"kubernetes.io/projected/0b1e7f49-1803-44d3-977b-5bcae4fe831b-kube-api-access-5sc9d\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.577382 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.577359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b1e7f49-1803-44d3-977b-5bcae4fe831b-manager-config\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.579534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.579510    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1e7f49-1803-44d3-977b-5bcae4fe831b-cert\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.579644 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.579507    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b1e7f49-1803-44d3-977b-5bcae4fe831b-metrics-cert\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.588325 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.588299    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc9d\" (UniqueName: \"kubernetes.io/projected/0b1e7f49-1803-44d3-977b-5bcae4fe831b-kube-api-access-5sc9d\") pod \"lws-controller-manager-db6444945-knhg9\" (UID: \"0b1e7f49-1803-44d3-977b-5bcae4fe831b\") " pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.762726 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.762646    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:49.892486 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:49.892444    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-db6444945-knhg9"]
Jul 09 16:55:49.895191 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:55:49.895167    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1e7f49_1803_44d3_977b_5bcae4fe831b.slice/crio-f7ff6d7b80e8cc0f02a838ac728b549ac5ed3dff5bf0c8ff10ff101c26458792 WatchSource:0}: Error finding container f7ff6d7b80e8cc0f02a838ac728b549ac5ed3dff5bf0c8ff10ff101c26458792: Status 404 returned error can't find the container with id f7ff6d7b80e8cc0f02a838ac728b549ac5ed3dff5bf0c8ff10ff101c26458792
Jul 09 16:55:50.095648 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:50.095613    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9" event={"ID":"0b1e7f49-1803-44d3-977b-5bcae4fe831b","Type":"ContainerStarted","Data":"f7ff6d7b80e8cc0f02a838ac728b549ac5ed3dff5bf0c8ff10ff101c26458792"}
Jul 09 16:55:53.107985 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:53.107945    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9" event={"ID":"0b1e7f49-1803-44d3-977b-5bcae4fe831b","Type":"ContainerStarted","Data":"bc4fdbcd25c572ddef2237085394c5e526cc1a373048807836557a6900b969c9"}
Jul 09 16:55:53.108448 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:53.108080    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:55:53.129635 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:53.129575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9" podStartSLOduration=1.927987238 podStartE2EDuration="4.129559062s" totalImagesPullingTime="2.201571824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:55:53.128101291 +0000 UTC m=+293.825273620" watchObservedRunningTime="2026-07-09 16:55:53.129559062 +0000 UTC m=+293.826731389"
Jul 09 16:55:59.719707 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:55:59.719634    2567 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 16:56:00.551535 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.551504    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"]
Jul 09 16:56:00.554535 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.554520    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.557342 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.557320    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 16:56:00.557557 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.557533    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 16:56:00.557692 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.557567    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sh896\""
Jul 09 16:56:00.557692 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.557598    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 16:56:00.557850 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.557837    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 16:56:00.572275 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.572250    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"]
Jul 09 16:56:00.661422 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.661401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/e71bbb6e-28e9-490c-b09b-0d5256f061bf-kube-api-access-kdshm\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.661534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.661429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e71bbb6e-28e9-490c-b09b-0d5256f061bf-manifests\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.661534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.661454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e71bbb6e-28e9-490c-b09b-0d5256f061bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.661534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.661478    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e71bbb6e-28e9-490c-b09b-0d5256f061bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.661534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.661533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e71bbb6e-28e9-490c-b09b-0d5256f061bf-tmp\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.761934 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.761899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/e71bbb6e-28e9-490c-b09b-0d5256f061bf-kube-api-access-kdshm\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.761934 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.761936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e71bbb6e-28e9-490c-b09b-0d5256f061bf-manifests\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.762652 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.761962    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e71bbb6e-28e9-490c-b09b-0d5256f061bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.762652 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.761994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e71bbb6e-28e9-490c-b09b-0d5256f061bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.762652 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.762040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e71bbb6e-28e9-490c-b09b-0d5256f061bf-tmp\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.762652 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.762447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e71bbb6e-28e9-490c-b09b-0d5256f061bf-manifests\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.764890 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.764857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e71bbb6e-28e9-490c-b09b-0d5256f061bf-tmp\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.765092 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.765057    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e71bbb6e-28e9-490c-b09b-0d5256f061bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.765269 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.765248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e71bbb6e-28e9-490c-b09b-0d5256f061bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.775399 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.775374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdshm\" (UniqueName: \"kubernetes.io/projected/e71bbb6e-28e9-490c-b09b-0d5256f061bf-kube-api-access-kdshm\") pod \"opendatahub-operator-controller-manager-789dddf866-mnvbm\" (UID: \"e71bbb6e-28e9-490c-b09b-0d5256f061bf\") " pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:00.865053 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:00.865027    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:01.017098 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:01.017071    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"]
Jul 09 16:56:01.019710 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:56:01.019686    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71bbb6e_28e9_490c_b09b_0d5256f061bf.slice/crio-2453ff5944441d9ab3c04c7c7bae5b6b74e853c16423ee433497bf7ef04c56e6 WatchSource:0}: Error finding container 2453ff5944441d9ab3c04c7c7bae5b6b74e853c16423ee433497bf7ef04c56e6: Status 404 returned error can't find the container with id 2453ff5944441d9ab3c04c7c7bae5b6b74e853c16423ee433497bf7ef04c56e6
Jul 09 16:56:01.138110 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:01.138049    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm" event={"ID":"e71bbb6e-28e9-490c-b09b-0d5256f061bf","Type":"ContainerStarted","Data":"2453ff5944441d9ab3c04c7c7bae5b6b74e853c16423ee433497bf7ef04c56e6"}
Jul 09 16:56:04.113912 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:04.113882    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-db6444945-knhg9"
Jul 09 16:56:05.156338 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:05.156298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm" event={"ID":"e71bbb6e-28e9-490c-b09b-0d5256f061bf","Type":"ContainerStarted","Data":"c44bedd5f8b7026aad95d898d306a0460176a1edcf11afe8594d85228bedb96e"}
Jul 09 16:56:06.161665 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:06.161624    2567 generic.go:356] "Generic (PLEG): container finished" podID="e71bbb6e-28e9-490c-b09b-0d5256f061bf" containerID="c44bedd5f8b7026aad95d898d306a0460176a1edcf11afe8594d85228bedb96e" exitCode=0
Jul 09 16:56:06.162136 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:06.161703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm" event={"ID":"e71bbb6e-28e9-490c-b09b-0d5256f061bf","Type":"ContainerDied","Data":"c44bedd5f8b7026aad95d898d306a0460176a1edcf11afe8594d85228bedb96e"}
Jul 09 16:56:06.162324 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:06.162304    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:56:07.167245 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:07.167212    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm" event={"ID":"e71bbb6e-28e9-490c-b09b-0d5256f061bf","Type":"ContainerStarted","Data":"c15a32aebb671ecf55ecda0207e8e0e5b7c1285ede25715cfce16af391082749"}
Jul 09 16:56:07.167582 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:07.167285    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:07.193070 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:07.192991    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm" podStartSLOduration=2.217386681 podStartE2EDuration="7.192976988s" totalImagesPullingTime="3.906652446s" totalInitContainerRuntime="1.068937861s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:07.191824244 +0000 UTC m=+307.888996571" watchObservedRunningTime="2026-07-09 16:56:07.192976988 +0000 UTC m=+307.890149315"
Jul 09 16:56:18.173355 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:18.173326    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-789dddf866-mnvbm"
Jul 09 16:56:20.561003 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.560970    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"]
Jul 09 16:56:20.564089 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.564074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.568761 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.568740    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 16:56:20.568918 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.568787    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 16:56:20.569858 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.569832    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 16:56:20.570034 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.569873    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v9qf2\""
Jul 09 16:56:20.570034 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.569908    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 16:56:20.588163 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.588140    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"]
Jul 09 16:56:20.711931 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.711905    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tls-certs\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.711931 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.711934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tmp\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.712140 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.711984    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpzf\" (UniqueName: \"kubernetes.io/projected/7108effc-c7bd-4c82-9a09-c3fa30ef8102-kube-api-access-9lpzf\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.812694 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.812623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpzf\" (UniqueName: \"kubernetes.io/projected/7108effc-c7bd-4c82-9a09-c3fa30ef8102-kube-api-access-9lpzf\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.812840 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.812703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tls-certs\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.812840 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.812732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tmp\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.812945 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:56:20.812904    2567 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 09 16:56:20.812989 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:56:20.812971    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tls-certs podName:7108effc-c7bd-4c82-9a09-c3fa30ef8102 nodeName:}" failed. No retries permitted until 2026-07-09 16:56:21.312951376 +0000 UTC m=+322.010123695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tls-certs") pod "kube-auth-proxy-cc96589f5-jcvjk" (UID: "7108effc-c7bd-4c82-9a09-c3fa30ef8102") : secret "kube-auth-proxy-tls" not found
Jul 09 16:56:20.815139 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.815116    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tmp\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:20.827299 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:20.827279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpzf\" (UniqueName: \"kubernetes.io/projected/7108effc-c7bd-4c82-9a09-c3fa30ef8102-kube-api-access-9lpzf\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:21.317187 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:21.317152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tls-certs\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:21.319710 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:21.319685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7108effc-c7bd-4c82-9a09-c3fa30ef8102-tls-certs\") pod \"kube-auth-proxy-cc96589f5-jcvjk\" (UID: \"7108effc-c7bd-4c82-9a09-c3fa30ef8102\") " pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:21.473900 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:21.473876    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"
Jul 09 16:56:21.606556 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:21.606533    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk"]
Jul 09 16:56:21.608450 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:56:21.608424    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7108effc_c7bd_4c82_9a09_c3fa30ef8102.slice/crio-0fa1646befcd81c731c7692522d2a796f677495db01d1ce6902a4d1534d6aa9b WatchSource:0}: Error finding container 0fa1646befcd81c731c7692522d2a796f677495db01d1ce6902a4d1534d6aa9b: Status 404 returned error can't find the container with id 0fa1646befcd81c731c7692522d2a796f677495db01d1ce6902a4d1534d6aa9b
Jul 09 16:56:22.220756 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:22.220722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk" event={"ID":"7108effc-c7bd-4c82-9a09-c3fa30ef8102","Type":"ContainerStarted","Data":"0fa1646befcd81c731c7692522d2a796f677495db01d1ce6902a4d1534d6aa9b"}
Jul 09 16:56:25.235397 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:25.235362    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk" event={"ID":"7108effc-c7bd-4c82-9a09-c3fa30ef8102","Type":"ContainerStarted","Data":"8a72c2f7cab83dee4f28bc5432e2c0171d415165f8fa8c4aea71df4bd19c46b4"}
Jul 09 16:56:25.254896 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:25.254853    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cc96589f5-jcvjk" podStartSLOduration=2.462635669 podStartE2EDuration="5.25483758s" totalImagesPullingTime="2.792201911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:25.253638432 +0000 UTC m=+325.950810759" watchObservedRunningTime="2026-07-09 16:56:25.25483758 +0000 UTC m=+325.952009906"
Jul 09 16:56:33.086534 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.086494    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"]
Jul 09 16:56:33.090113 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.090090    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"]
Jul 09 16:56:33.090258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.090235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.093098 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.093070    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mnw29\""
Jul 09 16:56:33.093498 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.093155    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 09 16:56:33.093498 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.093175    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.095962 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.095943    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 09 16:56:33.096081 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.096029    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2xt6j\""
Jul 09 16:56:33.103970 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.103950    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"]
Jul 09 16:56:33.108078 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.108055    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"]
Jul 09 16:56:33.212286 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.212264    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ss6c\" (UniqueName: \"kubernetes.io/projected/3f1d1892-8663-4908-b974-f70fc205b18f-kube-api-access-7ss6c\") pod \"kserve-controller-manager-d6f4b65b8-bjvcf\" (UID: \"3f1d1892-8663-4908-b974-f70fc205b18f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.212405 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.212304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f1d1892-8663-4908-b974-f70fc205b18f-cert\") pod \"kserve-controller-manager-d6f4b65b8-bjvcf\" (UID: \"3f1d1892-8663-4908-b974-f70fc205b18f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.212405 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.212349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d834298-f377-43c7-ade1-e5e12c818d78-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.212405 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.212375    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7qcs\" (UniqueName: \"kubernetes.io/projected/8d834298-f377-43c7-ade1-e5e12c818d78-kube-api-access-x7qcs\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.313095 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.313067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f1d1892-8663-4908-b974-f70fc205b18f-cert\") pod \"kserve-controller-manager-d6f4b65b8-bjvcf\" (UID: \"3f1d1892-8663-4908-b974-f70fc205b18f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.313241 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.313116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d834298-f377-43c7-ade1-e5e12c818d78-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.313241 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.313161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7qcs\" (UniqueName: \"kubernetes.io/projected/8d834298-f377-43c7-ade1-e5e12c818d78-kube-api-access-x7qcs\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.313241 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.313190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ss6c\" (UniqueName: \"kubernetes.io/projected/3f1d1892-8663-4908-b974-f70fc205b18f-kube-api-access-7ss6c\") pod \"kserve-controller-manager-d6f4b65b8-bjvcf\" (UID: \"3f1d1892-8663-4908-b974-f70fc205b18f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.313405 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:56:33.313313    2567 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 09 16:56:33.313405 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:56:33.313400    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d834298-f377-43c7-ade1-e5e12c818d78-cert podName:8d834298-f377-43c7-ade1-e5e12c818d78 nodeName:}" failed. No retries permitted until 2026-07-09 16:56:33.813379002 +0000 UTC m=+334.510551312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8d834298-f377-43c7-ade1-e5e12c818d78-cert") pod "llmisvc-controller-manager-777c6bd7d5-mvspw" (UID: "8d834298-f377-43c7-ade1-e5e12c818d78") : secret "llmisvc-webhook-server-cert" not found
Jul 09 16:56:33.315726 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.315697    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f1d1892-8663-4908-b974-f70fc205b18f-cert\") pod \"kserve-controller-manager-d6f4b65b8-bjvcf\" (UID: \"3f1d1892-8663-4908-b974-f70fc205b18f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.322882 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.322859    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ss6c\" (UniqueName: \"kubernetes.io/projected/3f1d1892-8663-4908-b974-f70fc205b18f-kube-api-access-7ss6c\") pod \"kserve-controller-manager-d6f4b65b8-bjvcf\" (UID: \"3f1d1892-8663-4908-b974-f70fc205b18f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.323292 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.323274    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7qcs\" (UniqueName: \"kubernetes.io/projected/8d834298-f377-43c7-ade1-e5e12c818d78-kube-api-access-x7qcs\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.403092 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.403069    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:33.531031 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.531000    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"]
Jul 09 16:56:33.533753 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:56:33.533724    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1d1892_8663_4908_b974_f70fc205b18f.slice/crio-da515b986807b342a6d58bf3c34baa7c4aee8aefa486bc46a7dee8d50876da4b WatchSource:0}: Error finding container da515b986807b342a6d58bf3c34baa7c4aee8aefa486bc46a7dee8d50876da4b: Status 404 returned error can't find the container with id da515b986807b342a6d58bf3c34baa7c4aee8aefa486bc46a7dee8d50876da4b
Jul 09 16:56:33.817354 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.817267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d834298-f377-43c7-ade1-e5e12c818d78-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:33.819984 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:33.819963    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d834298-f377-43c7-ade1-e5e12c818d78-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mvspw\" (UID: \"8d834298-f377-43c7-ade1-e5e12c818d78\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:34.010322 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:34.010289    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:34.140407 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:34.140353    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"]
Jul 09 16:56:34.143252 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:56:34.143227    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d834298_f377_43c7_ade1_e5e12c818d78.slice/crio-ece53931be19b8bc7e44591916f6ecdb438a04f54fd6b1704da1bb73c74450af WatchSource:0}: Error finding container ece53931be19b8bc7e44591916f6ecdb438a04f54fd6b1704da1bb73c74450af: Status 404 returned error can't find the container with id ece53931be19b8bc7e44591916f6ecdb438a04f54fd6b1704da1bb73c74450af
Jul 09 16:56:34.272299 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:34.272270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw" event={"ID":"8d834298-f377-43c7-ade1-e5e12c818d78","Type":"ContainerStarted","Data":"ece53931be19b8bc7e44591916f6ecdb438a04f54fd6b1704da1bb73c74450af"}
Jul 09 16:56:34.273382 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:34.273359    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf" event={"ID":"3f1d1892-8663-4908-b974-f70fc205b18f","Type":"ContainerStarted","Data":"da515b986807b342a6d58bf3c34baa7c4aee8aefa486bc46a7dee8d50876da4b"}
Jul 09 16:56:37.290226 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:37.290182    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf" event={"ID":"3f1d1892-8663-4908-b974-f70fc205b18f","Type":"ContainerStarted","Data":"a17b3c007ac4b8aa3d8d67e163c71f8cb00227820fd724e1afa9f2a6d33fa153"}
Jul 09 16:56:37.290654 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:37.290295    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:56:37.310137 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:37.310082    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf" podStartSLOduration=1.277124613 podStartE2EDuration="4.310065996s" totalImagesPullingTime="3.032941383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:37.308694337 +0000 UTC m=+338.005866665" watchObservedRunningTime="2026-07-09 16:56:37.310065996 +0000 UTC m=+338.007238325"
Jul 09 16:56:38.297181 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:38.297135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw" event={"ID":"8d834298-f377-43c7-ade1-e5e12c818d78","Type":"ContainerStarted","Data":"54cfd86fb6b97588cbc17725257602bf87644454487eba5d12ba2ee814bbf44d"}
Jul 09 16:56:38.297617 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:38.297246    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:56:38.324086 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:56:38.324037    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw" podStartSLOduration=2.006750269 podStartE2EDuration="5.324022239s" totalImagesPullingTime="3.31727197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:38.322914993 +0000 UTC m=+339.020087331" watchObservedRunningTime="2026-07-09 16:56:38.324022239 +0000 UTC m=+339.021194565"
Jul 09 16:57:08.302918 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:57:08.302885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bjvcf"
Jul 09 16:57:09.304460 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:57:09.304424    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mvspw"
Jul 09 16:58:08.711931 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.711889    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"]
Jul 09 16:58:08.715240 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.715220    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.719099 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.718394    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 09 16:58:08.719099 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.718479    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 16:58:08.719099 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.718690    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7ms9p\""
Jul 09 16:58:08.719099 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.718723    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 09 16:58:08.719352 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.719333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 16:58:08.730667 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.730644    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"]
Jul 09 16:58:08.787664 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.787635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0209346f-8206-4716-a6bf-d1631786923c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.787664 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.787668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzdd\" (UniqueName: \"kubernetes.io/projected/0209346f-8206-4716-a6bf-d1631786923c-kube-api-access-hjzdd\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.787910 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.787701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0209346f-8206-4716-a6bf-d1631786923c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.888667 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.888636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0209346f-8206-4716-a6bf-d1631786923c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.888667 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.888666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzdd\" (UniqueName: \"kubernetes.io/projected/0209346f-8206-4716-a6bf-d1631786923c-kube-api-access-hjzdd\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.888897 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.888690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0209346f-8206-4716-a6bf-d1631786923c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.888897 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:58:08.888785    2567 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 09 16:58:08.888897 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:58:08.888875    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0209346f-8206-4716-a6bf-d1631786923c-plugin-serving-cert podName:0209346f-8206-4716-a6bf-d1631786923c nodeName:}" failed. No retries permitted until 2026-07-09 16:58:09.388857084 +0000 UTC m=+430.086029390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0209346f-8206-4716-a6bf-d1631786923c-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-4s49b" (UID: "0209346f-8206-4716-a6bf-d1631786923c") : secret "plugin-serving-cert" not found
Jul 09 16:58:08.889255 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.889238    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0209346f-8206-4716-a6bf-d1631786923c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:08.899879 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:08.899859    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzdd\" (UniqueName: \"kubernetes.io/projected/0209346f-8206-4716-a6bf-d1631786923c-kube-api-access-hjzdd\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:09.393287 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:09.393252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0209346f-8206-4716-a6bf-d1631786923c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:09.395880 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:09.395855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0209346f-8206-4716-a6bf-d1631786923c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4s49b\" (UID: \"0209346f-8206-4716-a6bf-d1631786923c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:09.626270 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:09.626224    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"
Jul 09 16:58:09.762441 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:09.762414    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b"]
Jul 09 16:58:09.765072 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:58:09.765045    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0209346f_8206_4716_a6bf_d1631786923c.slice/crio-2dd789200f92b9b94f09cba37364135b5a1c010c6c76f25ec4a2f842b0323ea6 WatchSource:0}: Error finding container 2dd789200f92b9b94f09cba37364135b5a1c010c6c76f25ec4a2f842b0323ea6: Status 404 returned error can't find the container with id 2dd789200f92b9b94f09cba37364135b5a1c010c6c76f25ec4a2f842b0323ea6
Jul 09 16:58:10.688938 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:10.688897    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b" event={"ID":"0209346f-8206-4716-a6bf-d1631786923c","Type":"ContainerStarted","Data":"2dd789200f92b9b94f09cba37364135b5a1c010c6c76f25ec4a2f842b0323ea6"}
Jul 09 16:58:32.800504 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:32.800465    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b" event={"ID":"0209346f-8206-4716-a6bf-d1631786923c","Type":"ContainerStarted","Data":"85243c8149836fd627acdf741789f1e973b24f6aef917659a592df90df08a4d1"}
Jul 09 16:58:32.818168 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:32.818023    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4s49b" podStartSLOduration=2.723530262 podStartE2EDuration="24.818006093s" totalImagesPullingTime="22.094475831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:58:32.817709071 +0000 UTC m=+453.514881391" watchObservedRunningTime="2026-07-09 16:58:32.818006093 +0000 UTC m=+453.515178421"
Jul 09 16:58:50.746187 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.746147    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 16:58:50.749339 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.749321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:50.752328 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.752307    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 16:58:50.772007 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.771986    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 16:58:50.825881 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.825847    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 16:58:50.851941 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.851913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzg87\" (UniqueName: \"kubernetes.io/projected/6c9bed73-2881-4ba0-bc0f-d759bab1e883-kube-api-access-xzg87\") pod \"limitador-limitador-54d6d988d5-fcks8\" (UID: \"6c9bed73-2881-4ba0-bc0f-d759bab1e883\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:50.852073 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.851957    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6c9bed73-2881-4ba0-bc0f-d759bab1e883-config-file\") pod \"limitador-limitador-54d6d988d5-fcks8\" (UID: \"6c9bed73-2881-4ba0-bc0f-d759bab1e883\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:50.952628 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.952597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzg87\" (UniqueName: \"kubernetes.io/projected/6c9bed73-2881-4ba0-bc0f-d759bab1e883-kube-api-access-xzg87\") pod \"limitador-limitador-54d6d988d5-fcks8\" (UID: \"6c9bed73-2881-4ba0-bc0f-d759bab1e883\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:50.952759 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.952659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6c9bed73-2881-4ba0-bc0f-d759bab1e883-config-file\") pod \"limitador-limitador-54d6d988d5-fcks8\" (UID: \"6c9bed73-2881-4ba0-bc0f-d759bab1e883\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:50.953259 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.953240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6c9bed73-2881-4ba0-bc0f-d759bab1e883-config-file\") pod \"limitador-limitador-54d6d988d5-fcks8\" (UID: \"6c9bed73-2881-4ba0-bc0f-d759bab1e883\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:50.960922 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:50.960901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzg87\" (UniqueName: \"kubernetes.io/projected/6c9bed73-2881-4ba0-bc0f-d759bab1e883-kube-api-access-xzg87\") pod \"limitador-limitador-54d6d988d5-fcks8\" (UID: \"6c9bed73-2881-4ba0-bc0f-d759bab1e883\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:51.059231 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.059153    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:51.185686 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.185630    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 16:58:51.187936 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:58:51.187898    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9bed73_2881_4ba0_bc0f_d759bab1e883.slice/crio-894b775cbcb218951704f2663b79f7b57ebe4c7ad59b6928f69cc997019ea8fe WatchSource:0}: Error finding container 894b775cbcb218951704f2663b79f7b57ebe4c7ad59b6928f69cc997019ea8fe: Status 404 returned error can't find the container with id 894b775cbcb218951704f2663b79f7b57ebe4c7ad59b6928f69cc997019ea8fe
Jul 09 16:58:51.300247 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.300222    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-zhgr4"]
Jul 09 16:58:51.303124 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.303109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:58:51.306319 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.306298    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nt42k\""
Jul 09 16:58:51.310299 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.310273    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zhgr4"]
Jul 09 16:58:51.355623 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.355590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmzp\" (UniqueName: \"kubernetes.io/projected/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0-kube-api-access-dkmzp\") pod \"authorino-75d556bd77-zhgr4\" (UID: \"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0\") " pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:58:51.456532 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.456505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmzp\" (UniqueName: \"kubernetes.io/projected/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0-kube-api-access-dkmzp\") pod \"authorino-75d556bd77-zhgr4\" (UID: \"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0\") " pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:58:51.467575 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.467546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmzp\" (UniqueName: \"kubernetes.io/projected/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0-kube-api-access-dkmzp\") pod \"authorino-75d556bd77-zhgr4\" (UID: \"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0\") " pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:58:51.613183 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.613158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:58:51.770462 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.770403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zhgr4"]
Jul 09 16:58:51.775221 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:58:51.775186    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda44bf19d_3e56_4a51_ab7b_f524e7af7fc0.slice/crio-a3934a88433e6ebbe28df4340d733350620630443f1c613923df03625a099647 WatchSource:0}: Error finding container a3934a88433e6ebbe28df4340d733350620630443f1c613923df03625a099647: Status 404 returned error can't find the container with id a3934a88433e6ebbe28df4340d733350620630443f1c613923df03625a099647
Jul 09 16:58:51.889201 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.889125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zhgr4" event={"ID":"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0","Type":"ContainerStarted","Data":"a3934a88433e6ebbe28df4340d733350620630443f1c613923df03625a099647"}
Jul 09 16:58:51.891654 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:51.891623    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8" event={"ID":"6c9bed73-2881-4ba0-bc0f-d759bab1e883","Type":"ContainerStarted","Data":"894b775cbcb218951704f2663b79f7b57ebe4c7ad59b6928f69cc997019ea8fe"}
Jul 09 16:58:55.914405 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:55.914324    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8" event={"ID":"6c9bed73-2881-4ba0-bc0f-d759bab1e883","Type":"ContainerStarted","Data":"ea738121f5c543e4d20574c81f6032a1025d8af8a0081ecf9187ae5deb26dae5"}
Jul 09 16:58:55.914405 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:55.914397    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:58:55.915946 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:55.915921    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zhgr4" event={"ID":"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0","Type":"ContainerStarted","Data":"dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84"}
Jul 09 16:58:55.930138 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:55.930079    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8" podStartSLOduration=1.472417057 podStartE2EDuration="5.930063724s" totalImagesPullingTime="4.457646667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:58:55.929494973 +0000 UTC m=+476.626667300" watchObservedRunningTime="2026-07-09 16:58:55.930063724 +0000 UTC m=+476.627236046"
Jul 09 16:58:55.942641 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:58:55.942593    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-zhgr4" podStartSLOduration=1.066040233 podStartE2EDuration="4.942578345s" totalImagesPullingTime="3.876538112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:58:55.942317475 +0000 UTC m=+476.639489808" watchObservedRunningTime="2026-07-09 16:58:55.942578345 +0000 UTC m=+476.639750673"
Jul 09 16:59:06.926935 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:06.926896    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcks8"
Jul 09 16:59:24.895531 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:24.895499    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58d448d654-h9z9m"]
Jul 09 16:59:24.902592 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:24.902575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:24.905622 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:24.905595    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 16:59:24.905919 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:24.905896    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58d448d654-h9z9m"]
Jul 09 16:59:25.009146 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.009122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a96916b2-deb7-4665-bcba-b83cd1a2461a-tls-cert\") pod \"authorino-58d448d654-h9z9m\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") " pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.009282 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.009157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqn7\" (UniqueName: \"kubernetes.io/projected/a96916b2-deb7-4665-bcba-b83cd1a2461a-kube-api-access-9gqn7\") pod \"authorino-58d448d654-h9z9m\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") " pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.110313 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.110289    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a96916b2-deb7-4665-bcba-b83cd1a2461a-tls-cert\") pod \"authorino-58d448d654-h9z9m\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") " pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.110423 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.110325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqn7\" (UniqueName: \"kubernetes.io/projected/a96916b2-deb7-4665-bcba-b83cd1a2461a-kube-api-access-9gqn7\") pod \"authorino-58d448d654-h9z9m\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") " pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.112698 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.112676    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a96916b2-deb7-4665-bcba-b83cd1a2461a-tls-cert\") pod \"authorino-58d448d654-h9z9m\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") " pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.117696 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.117672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqn7\" (UniqueName: \"kubernetes.io/projected/a96916b2-deb7-4665-bcba-b83cd1a2461a-kube-api-access-9gqn7\") pod \"authorino-58d448d654-h9z9m\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") " pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.212070 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.212010    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 16:59:25.336784 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:25.336757    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58d448d654-h9z9m"]
Jul 09 16:59:25.338733 ip-10-0-128-131 kubenswrapper[2567]: W0709 16:59:25.338708    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96916b2_deb7_4665_bcba_b83cd1a2461a.slice/crio-de259da8ada4d238e1f7878b2b6a19b78593fe535f69be7cdde1cfc071216940 WatchSource:0}: Error finding container de259da8ada4d238e1f7878b2b6a19b78593fe535f69be7cdde1cfc071216940: Status 404 returned error can't find the container with id de259da8ada4d238e1f7878b2b6a19b78593fe535f69be7cdde1cfc071216940
Jul 09 16:59:26.051690 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.051655    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d448d654-h9z9m" event={"ID":"a96916b2-deb7-4665-bcba-b83cd1a2461a","Type":"ContainerStarted","Data":"b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22"}
Jul 09 16:59:26.052058 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.051698    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d448d654-h9z9m" event={"ID":"a96916b2-deb7-4665-bcba-b83cd1a2461a","Type":"ContainerStarted","Data":"de259da8ada4d238e1f7878b2b6a19b78593fe535f69be7cdde1cfc071216940"}
Jul 09 16:59:26.068490 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.068383    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58d448d654-h9z9m" podStartSLOduration=1.590705363 podStartE2EDuration="2.06836918s" totalImagesPullingTime="477.663817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:59:26.067837532 +0000 UTC m=+506.765009860" watchObservedRunningTime="2026-07-09 16:59:26.06836918 +0000 UTC m=+506.765541551"
Jul 09 16:59:26.091644 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.091612    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zhgr4"]
Jul 09 16:59:26.091832 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.091807    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-zhgr4" podUID="a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" containerName="authorino" containerID="cri-o://dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84" gracePeriod=30
Jul 09 16:59:26.329714 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.329666    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:59:26.422105 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.422080    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkmzp\" (UniqueName: \"kubernetes.io/projected/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0-kube-api-access-dkmzp\") pod \"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0\" (UID: \"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0\") "
Jul 09 16:59:26.424395 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.424358    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0-kube-api-access-dkmzp" (OuterVolumeSpecName: "kube-api-access-dkmzp") pod "a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" (UID: "a44bf19d-3e56-4a51-ab7b-f524e7af7fc0"). InnerVolumeSpecName "kube-api-access-dkmzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:59:26.523210 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:26.523185    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkmzp\" (UniqueName: \"kubernetes.io/projected/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0-kube-api-access-dkmzp\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 16:59:27.063505 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.063471    2567 generic.go:356] "Generic (PLEG): container finished" podID="a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" containerID="dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84" exitCode=0
Jul 09 16:59:27.063964 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.063524    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zhgr4"
Jul 09 16:59:27.063964 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.063560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zhgr4" event={"ID":"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0","Type":"ContainerDied","Data":"dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84"}
Jul 09 16:59:27.063964 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.063615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zhgr4" event={"ID":"a44bf19d-3e56-4a51-ab7b-f524e7af7fc0","Type":"ContainerDied","Data":"a3934a88433e6ebbe28df4340d733350620630443f1c613923df03625a099647"}
Jul 09 16:59:27.063964 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.063655    2567 scope.go:117] "RemoveContainer" containerID="dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84"
Jul 09 16:59:27.072313 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.072297    2567 scope.go:117] "RemoveContainer" containerID="dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84"
Jul 09 16:59:27.072560 ip-10-0-128-131 kubenswrapper[2567]: E0709 16:59:27.072543    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84\": container with ID starting with dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84 not found: ID does not exist" containerID="dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84"
Jul 09 16:59:27.072608 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.072565    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84"} err="failed to get container status \"dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84\": rpc error: code = NotFound desc = could not find container \"dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84\": container with ID starting with dc1a6e6c9c5c2177fdd2349d6c69575c1751eb5f12ea312e092041a14ffeeb84 not found: ID does not exist"
Jul 09 16:59:27.083258 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.083237    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zhgr4"]
Jul 09 16:59:27.087156 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.087135    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zhgr4"]
Jul 09 16:59:27.872645 ip-10-0-128-131 kubenswrapper[2567]: I0709 16:59:27.872612    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" path="/var/lib/kubelet/pods/a44bf19d-3e56-4a51-ab7b-f524e7af7fc0/volumes"
Jul 09 17:00:07.375212 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:00:07.375180    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:00:24.658289 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:00:24.658251    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:00:30.062607 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:00:30.062567    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:00:39.663646 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:00:39.663609    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:00:55.257046 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:00:55.257000    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:01:01.161684 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:01.161642    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:01:16.144507 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.144463    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"]
Jul 09 17:01:16.145009 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.144808    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" containerName="authorino"
Jul 09 17:01:16.145009 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.144821    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" containerName="authorino"
Jul 09 17:01:16.145009 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.144875    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44bf19d-3e56-4a51-ab7b-f524e7af7fc0" containerName="authorino"
Jul 09 17:01:16.147643 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.147623    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.151868 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.151848    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 17:01:16.151975 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.151875    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 17:01:16.151975 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.151885    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 09 17:01:16.151975 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.151875    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sl724\""
Jul 09 17:01:16.157996 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.157973    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"]
Jul 09 17:01:16.259380 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.259356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.259518 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.259385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.259518 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.259413    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.259518 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.259460    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.259678 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.259540    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsdf\" (UniqueName: \"kubernetes.io/projected/c2257769-7410-4250-b057-be64c8bb78d2-kube-api-access-rmsdf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.259678 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.259559    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2257769-7410-4250-b057-be64c8bb78d2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360526 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360647 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360647 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360577    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360647 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360647 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsdf\" (UniqueName: \"kubernetes.io/projected/c2257769-7410-4250-b057-be64c8bb78d2-kube-api-access-rmsdf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360843 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2257769-7410-4250-b057-be64c8bb78d2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.360938 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.361002 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.360984    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.361212 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.361195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.363150 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.363128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2257769-7410-4250-b057-be64c8bb78d2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.363634 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.363589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2257769-7410-4250-b057-be64c8bb78d2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.368660 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.368632    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsdf\" (UniqueName: \"kubernetes.io/projected/c2257769-7410-4250-b057-be64c8bb78d2-kube-api-access-rmsdf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg\" (UID: \"c2257769-7410-4250-b057-be64c8bb78d2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.457011 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.456953    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:16.588187 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.588133    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"]
Jul 09 17:01:16.590861 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:01:16.590831    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2257769_7410_4250_b057_be64c8bb78d2.slice/crio-bb157833916823d7808a314d4ccf3295c4e0fc452d781573c8c1588b8bbefa97 WatchSource:0}: Error finding container bb157833916823d7808a314d4ccf3295c4e0fc452d781573c8c1588b8bbefa97: Status 404 returned error can't find the container with id bb157833916823d7808a314d4ccf3295c4e0fc452d781573c8c1588b8bbefa97
Jul 09 17:01:16.592454 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:16.592429    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:01:17.545705 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:17.545658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg" event={"ID":"c2257769-7410-4250-b057-be64c8bb78d2","Type":"ContainerStarted","Data":"bb157833916823d7808a314d4ccf3295c4e0fc452d781573c8c1588b8bbefa97"}
Jul 09 17:01:18.552341 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:18.552307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg" event={"ID":"c2257769-7410-4250-b057-be64c8bb78d2","Type":"ContainerStarted","Data":"edd49c4fd0aededa170316e41650a1112b6bebaa618d08fe99427f90a41b6657"}
Jul 09 17:01:18.566496 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:18.566467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:01:18.571576 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:18.571535    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg" podStartSLOduration=1.3313323559999999 podStartE2EDuration="2.571519825s" totalImagesPullingTime="1.240187469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:01:18.570988642 +0000 UTC m=+619.268160969" watchObservedRunningTime="2026-07-09 17:01:18.571519825 +0000 UTC m=+619.268692151"
Jul 09 17:01:26.457319 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:26.457228    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:26.457319 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:26.457288    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:26.469946 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:26.469915    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:01:26.597295 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:01:26.597272    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg"
Jul 09 17:02:04.866177 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:04.866138    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f57fdb9f-hw4v8"]
Jul 09 17:02:04.868859 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:04.868823    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:04.876490 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:04.876466    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f57fdb9f-hw4v8"]
Jul 09 17:02:04.951551 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:04.951513    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/060e6d2b-1b47-42b4-a547-953cacc3b41f-tls-cert\") pod \"authorino-7f57fdb9f-hw4v8\" (UID: \"060e6d2b-1b47-42b4-a547-953cacc3b41f\") " pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:04.951712 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:04.951613    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fv69\" (UniqueName: \"kubernetes.io/projected/060e6d2b-1b47-42b4-a547-953cacc3b41f-kube-api-access-8fv69\") pod \"authorino-7f57fdb9f-hw4v8\" (UID: \"060e6d2b-1b47-42b4-a547-953cacc3b41f\") " pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:05.052692 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.052663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/060e6d2b-1b47-42b4-a547-953cacc3b41f-tls-cert\") pod \"authorino-7f57fdb9f-hw4v8\" (UID: \"060e6d2b-1b47-42b4-a547-953cacc3b41f\") " pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:05.052823 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.052727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fv69\" (UniqueName: \"kubernetes.io/projected/060e6d2b-1b47-42b4-a547-953cacc3b41f-kube-api-access-8fv69\") pod \"authorino-7f57fdb9f-hw4v8\" (UID: \"060e6d2b-1b47-42b4-a547-953cacc3b41f\") " pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:05.055152 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.055126    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/060e6d2b-1b47-42b4-a547-953cacc3b41f-tls-cert\") pod \"authorino-7f57fdb9f-hw4v8\" (UID: \"060e6d2b-1b47-42b4-a547-953cacc3b41f\") " pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:05.060653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.060634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fv69\" (UniqueName: \"kubernetes.io/projected/060e6d2b-1b47-42b4-a547-953cacc3b41f-kube-api-access-8fv69\") pod \"authorino-7f57fdb9f-hw4v8\" (UID: \"060e6d2b-1b47-42b4-a547-953cacc3b41f\") " pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:05.178189 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.178136    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f57fdb9f-hw4v8"
Jul 09 17:02:05.305685 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.305657    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f57fdb9f-hw4v8"]
Jul 09 17:02:05.308409 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:02:05.308383    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod060e6d2b_1b47_42b4_a547_953cacc3b41f.slice/crio-ad21f07fadc40d14175800283d151e7270d1185750c94299e15112d76158e45c WatchSource:0}: Error finding container ad21f07fadc40d14175800283d151e7270d1185750c94299e15112d76158e45c: Status 404 returned error can't find the container with id ad21f07fadc40d14175800283d151e7270d1185750c94299e15112d76158e45c
Jul 09 17:02:05.761001 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:05.760968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f57fdb9f-hw4v8" event={"ID":"060e6d2b-1b47-42b4-a547-953cacc3b41f","Type":"ContainerStarted","Data":"ad21f07fadc40d14175800283d151e7270d1185750c94299e15112d76158e45c"}
Jul 09 17:02:06.767714 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:06.767667    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f57fdb9f-hw4v8" event={"ID":"060e6d2b-1b47-42b4-a547-953cacc3b41f","Type":"ContainerStarted","Data":"bff13113131b98b19bbd6625da88e6f86be0d02db88795d5659f19e1e69ec4a8"}
Jul 09 17:02:06.783716 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:06.783663    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f57fdb9f-hw4v8" podStartSLOduration=2.339357219 podStartE2EDuration="2.783647061s" totalImagesPullingTime="444.289842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:02:06.78325772 +0000 UTC m=+667.480430050" watchObservedRunningTime="2026-07-09 17:02:06.783647061 +0000 UTC m=+667.480819390"
Jul 09 17:02:06.817633 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:06.817598    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58d448d654-h9z9m"]
Jul 09 17:02:06.817877 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:06.817845    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58d448d654-h9z9m" podUID="a96916b2-deb7-4665-bcba-b83cd1a2461a" containerName="authorino" containerID="cri-o://b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22" gracePeriod=30
Jul 09 17:02:07.071418 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.071372    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 17:02:07.173669 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.173113    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gqn7\" (UniqueName: \"kubernetes.io/projected/a96916b2-deb7-4665-bcba-b83cd1a2461a-kube-api-access-9gqn7\") pod \"a96916b2-deb7-4665-bcba-b83cd1a2461a\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") "
Jul 09 17:02:07.173669 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.173207    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a96916b2-deb7-4665-bcba-b83cd1a2461a-tls-cert\") pod \"a96916b2-deb7-4665-bcba-b83cd1a2461a\" (UID: \"a96916b2-deb7-4665-bcba-b83cd1a2461a\") "
Jul 09 17:02:07.189697 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.189659    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96916b2-deb7-4665-bcba-b83cd1a2461a-kube-api-access-9gqn7" (OuterVolumeSpecName: "kube-api-access-9gqn7") pod "a96916b2-deb7-4665-bcba-b83cd1a2461a" (UID: "a96916b2-deb7-4665-bcba-b83cd1a2461a"). InnerVolumeSpecName "kube-api-access-9gqn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:02:07.197233 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.197203    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96916b2-deb7-4665-bcba-b83cd1a2461a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a96916b2-deb7-4665-bcba-b83cd1a2461a" (UID: "a96916b2-deb7-4665-bcba-b83cd1a2461a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:02:07.274374 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.274336    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqn7\" (UniqueName: \"kubernetes.io/projected/a96916b2-deb7-4665-bcba-b83cd1a2461a-kube-api-access-9gqn7\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:02:07.274374 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.274371    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a96916b2-deb7-4665-bcba-b83cd1a2461a-tls-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:02:07.773458 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.773420    2567 generic.go:356] "Generic (PLEG): container finished" podID="a96916b2-deb7-4665-bcba-b83cd1a2461a" containerID="b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22" exitCode=0
Jul 09 17:02:07.773952 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.773482    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58d448d654-h9z9m"
Jul 09 17:02:07.773952 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.773504    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d448d654-h9z9m" event={"ID":"a96916b2-deb7-4665-bcba-b83cd1a2461a","Type":"ContainerDied","Data":"b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22"}
Jul 09 17:02:07.773952 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.773551    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58d448d654-h9z9m" event={"ID":"a96916b2-deb7-4665-bcba-b83cd1a2461a","Type":"ContainerDied","Data":"de259da8ada4d238e1f7878b2b6a19b78593fe535f69be7cdde1cfc071216940"}
Jul 09 17:02:07.773952 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.773569    2567 scope.go:117] "RemoveContainer" containerID="b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22"
Jul 09 17:02:07.782528 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.782512    2567 scope.go:117] "RemoveContainer" containerID="b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22"
Jul 09 17:02:07.782844 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:02:07.782822    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22\": container with ID starting with b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22 not found: ID does not exist" containerID="b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22"
Jul 09 17:02:07.782930 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.782856    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22"} err="failed to get container status \"b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22\": rpc error: code = NotFound desc = could not find container \"b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22\": container with ID starting with b313593a8d0573ff88304f7e288f7facbf117e587355bb3c701d9bbbe5a96d22 not found: ID does not exist"
Jul 09 17:02:07.796915 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.796889    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58d448d654-h9z9m"]
Jul 09 17:02:07.800717 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.800695    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58d448d654-h9z9m"]
Jul 09 17:02:07.872558 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:07.872521    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96916b2-deb7-4665-bcba-b83cd1a2461a" path="/var/lib/kubelet/pods/a96916b2-deb7-4665-bcba-b83cd1a2461a/volumes"
Jul 09 17:02:43.562737 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:43.562699    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:02:53.750826 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:02:53.750725    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:03:04.160154 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:03:04.160124    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:03:13.561700 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:03:13.561665    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:03:25.056706 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:03:25.056663    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:03:50.452622 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:03:50.452578    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:04:01.759435 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:04:01.759403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:04:34.962498 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:04:34.962464    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:04:45.256969 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:04:45.256925    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:04:53.667173 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:04:53.667130    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:05:03.056169 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:05:03.056131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:05:13.969475 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:05:13.969437    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:05:25.816238 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:05:25.816198    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:05:36.855524 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:05:36.855490    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:05:46.753820 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:05:46.753773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:06:12.057470 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:06:12.057434    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:06:23.466366 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:06:23.466326    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:06:46.766769 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:06:46.766734    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:06:57.261733 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:06:57.261696    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:07:12.560666 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:07:12.560629    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:07:21.753511 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:07:21.753469    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:07:32.465147 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:07:32.465109    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:07:42.057318 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:07:42.057284    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:07:51.358125 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:07:51.358082    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:09:38.258444 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:09:38.258408    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:09:49.463039 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:09:49.463004    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:09:59.866137 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:09:59.866101    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:10:10.458997 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:10:10.458963    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:10:22.259602 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:10:22.259511    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:10:32.355447 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:10:32.355410    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:10:41.456284 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:10:41.456248    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:10:52.452192 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:10:52.452157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:11:03.367201 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:11:03.367163    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:11:12.053718 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:11:12.053684    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:11:26.560419 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:11:26.560372    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:11:43.366729 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:11:43.366697    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:11:52.061633 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:11:52.061544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:12:03.254419 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:12:03.254381    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:12:12.461894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:12:12.461853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:12:28.363115 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:12:28.363072    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:12:37.766127 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:12:37.766092    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:12:47.360473 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:12:47.360436    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:12:54.919507 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:12:54.919471    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:13:05.158359 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:13:05.158311    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:13:14.355351 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:13:14.355318    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:13:27.967688 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:13:27.967613    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:13:37.271285 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:13:37.271249    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:13:46.859507 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:13:46.859467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:13:57.359340 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:13:57.359309    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:14:10.262674 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:14:10.262633    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:14:19.359931 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:14:19.359897    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:14:32.053272 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:14:32.053239    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:14:44.162738 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:14:44.162684    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:14:57.260388 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:14:57.260308    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:15:06.453000 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:06.452961    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:15:16.860396 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:16.860353    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:15:25.453617 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:25.453576    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:15:33.775467 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.775428    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"]
Jul 09 17:15:33.776012 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.775760    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96916b2-deb7-4665-bcba-b83cd1a2461a" containerName="authorino"
Jul 09 17:15:33.776012 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.775771    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96916b2-deb7-4665-bcba-b83cd1a2461a" containerName="authorino"
Jul 09 17:15:33.776012 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.775851    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96916b2-deb7-4665-bcba-b83cd1a2461a" containerName="authorino"
Jul 09 17:15:33.778730 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.778711    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:33.781539 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.781518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-720e00d5-serving-cert\""
Jul 09 17:15:33.781660 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.781592    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:15:33.781785 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.781767    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:15:33.782754 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.782737    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-whstr\""
Jul 09 17:15:33.789893 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.789870    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"]
Jul 09 17:15:33.903115 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.903085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr88q\" (UniqueName: \"kubernetes.io/projected/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-kube-api-access-nr88q\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:33.903254 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.903121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-maas-api-tls\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:33.903254 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:33.903143    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-openshift-service-ca\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.004171 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.004135    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr88q\" (UniqueName: \"kubernetes.io/projected/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-kube-api-access-nr88q\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.004292 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.004175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-maas-api-tls\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.004292 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.004203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-openshift-service-ca\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.004887 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.004868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-openshift-service-ca\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.006463 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.006444    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-maas-api-tls\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.014235 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.014216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr88q\" (UniqueName: \"kubernetes.io/projected/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-kube-api-access-nr88q\") pod \"maas-api-e2e-ait-720e00d5-54b54d787d-w9825\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.091078 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.091018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:34.226010 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.225973    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"]
Jul 09 17:15:34.229695 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:15:34.229669    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6a0b7d_bbfa_42ad_950c_80b250e6c221.slice/crio-c7d41e1ee50904a49d4fdc0af188eaf02ace65c9751dd482b40450cd650d40c1 WatchSource:0}: Error finding container c7d41e1ee50904a49d4fdc0af188eaf02ace65c9751dd482b40450cd650d40c1: Status 404 returned error can't find the container with id c7d41e1ee50904a49d4fdc0af188eaf02ace65c9751dd482b40450cd650d40c1
Jul 09 17:15:34.231366 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.231345    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:15:34.343851 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:34.343760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825" event={"ID":"cc6a0b7d-bbfa-42ad-950c-80b250e6c221","Type":"ContainerStarted","Data":"c7d41e1ee50904a49d4fdc0af188eaf02ace65c9751dd482b40450cd650d40c1"}
Jul 09 17:15:37.360283 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:37.360232    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825" event={"ID":"cc6a0b7d-bbfa-42ad-950c-80b250e6c221","Type":"ContainerStarted","Data":"426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40"}
Jul 09 17:15:37.360773 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:37.360327    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:37.377982 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:37.377926    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825" podStartSLOduration=2.153983995 podStartE2EDuration="4.377908481s" totalImagesPullingTime="2.223924486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:15:37.376625606 +0000 UTC m=+1478.073797934" watchObservedRunningTime="2026-07-09 17:15:37.377908481 +0000 UTC m=+1478.075080807"
Jul 09 17:15:39.023846 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.023815    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"]
Jul 09 17:15:39.369414 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.369373    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825" podUID="cc6a0b7d-bbfa-42ad-950c-80b250e6c221" containerName="maas-api" containerID="cri-o://426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40" gracePeriod=30
Jul 09 17:15:39.613870 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.613844    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:39.751472 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.751411    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr88q\" (UniqueName: \"kubernetes.io/projected/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-kube-api-access-nr88q\") pod \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") "
Jul 09 17:15:39.751472 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.751466    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-openshift-service-ca\") pod \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") "
Jul 09 17:15:39.751625 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.751548    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-maas-api-tls\") pod \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\" (UID: \"cc6a0b7d-bbfa-42ad-950c-80b250e6c221\") "
Jul 09 17:15:39.751894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.751867    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc6a0b7d-bbfa-42ad-950c-80b250e6c221" (UID: "cc6a0b7d-bbfa-42ad-950c-80b250e6c221"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:15:39.753653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.753632    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-kube-api-access-nr88q" (OuterVolumeSpecName: "kube-api-access-nr88q") pod "cc6a0b7d-bbfa-42ad-950c-80b250e6c221" (UID: "cc6a0b7d-bbfa-42ad-950c-80b250e6c221"). InnerVolumeSpecName "kube-api-access-nr88q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:15:39.753729 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.753709    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc6a0b7d-bbfa-42ad-950c-80b250e6c221" (UID: "cc6a0b7d-bbfa-42ad-950c-80b250e6c221"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:15:39.852241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.852218    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:15:39.852241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.852239    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr88q\" (UniqueName: \"kubernetes.io/projected/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-kube-api-access-nr88q\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:15:39.852365 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:39.852249    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6a0b7d-bbfa-42ad-950c-80b250e6c221-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:15:40.374761 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.374716    2567 generic.go:356] "Generic (PLEG): container finished" podID="cc6a0b7d-bbfa-42ad-950c-80b250e6c221" containerID="426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40" exitCode=0
Jul 09 17:15:40.374761 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.374756    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825" event={"ID":"cc6a0b7d-bbfa-42ad-950c-80b250e6c221","Type":"ContainerDied","Data":"426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40"}
Jul 09 17:15:40.375309 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.374774    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"
Jul 09 17:15:40.375309 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.374857    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825" event={"ID":"cc6a0b7d-bbfa-42ad-950c-80b250e6c221","Type":"ContainerDied","Data":"c7d41e1ee50904a49d4fdc0af188eaf02ace65c9751dd482b40450cd650d40c1"}
Jul 09 17:15:40.375309 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.374891    2567 scope.go:117] "RemoveContainer" containerID="426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40"
Jul 09 17:15:40.383150 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.383112    2567 scope.go:117] "RemoveContainer" containerID="426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40"
Jul 09 17:15:40.383402 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:15:40.383385    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40\": container with ID starting with 426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40 not found: ID does not exist" containerID="426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40"
Jul 09 17:15:40.383456 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.383409    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40"} err="failed to get container status \"426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40\": rpc error: code = NotFound desc = could not find container \"426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40\": container with ID starting with 426807cd556f051eeb98e7b18e08b2818f25a376b5f7ee19af70695c066acb40 not found: ID does not exist"
Jul 09 17:15:40.391534 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.391510    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"]
Jul 09 17:15:40.395787 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:40.395768    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-720e00d5-54b54d787d-w9825"]
Jul 09 17:15:41.871632 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:41.871600    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6a0b7d-bbfa-42ad-950c-80b250e6c221" path="/var/lib/kubelet/pods/cc6a0b7d-bbfa-42ad-950c-80b250e6c221/volumes"
Jul 09 17:15:48.758173 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:48.758099    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:15:59.760898 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:15:59.760859    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:17:37.757367 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:37.757326    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:17:45.356948 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:45.356907    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:17:54.067019 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.066943    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"]
Jul 09 17:17:54.067350 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.067313    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6a0b7d-bbfa-42ad-950c-80b250e6c221" containerName="maas-api"
Jul 09 17:17:54.067350 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.067325    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6a0b7d-bbfa-42ad-950c-80b250e6c221" containerName="maas-api"
Jul 09 17:17:54.067426 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.067390    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6a0b7d-bbfa-42ad-950c-80b250e6c221" containerName="maas-api"
Jul 09 17:17:54.070569 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.070547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.073213 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.073180    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-add69b2a-serving-cert\""
Jul 09 17:17:54.073322 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.073185    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:17:54.074436 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.074415    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:17:54.074547 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.074467    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-whstr\""
Jul 09 17:17:54.081423 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.081400    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"]
Jul 09 17:17:54.229774 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.229754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e856ec2-c462-4dad-a4ab-eb69460917e3-openshift-service-ca\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.229940 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.229791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.229940 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.229906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw92\" (UniqueName: \"kubernetes.io/projected/8e856ec2-c462-4dad-a4ab-eb69460917e3-kube-api-access-gvw92\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.331251 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.331185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw92\" (UniqueName: \"kubernetes.io/projected/8e856ec2-c462-4dad-a4ab-eb69460917e3-kube-api-access-gvw92\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.331369 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.331257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e856ec2-c462-4dad-a4ab-eb69460917e3-openshift-service-ca\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.331369 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.331297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.331497 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:17:54.331409    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-serving-cert: secret "maas-api-e2e-mt-add69b2a-serving-cert" not found
Jul 09 17:17:54.331497 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:17:54.331488    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls podName:8e856ec2-c462-4dad-a4ab-eb69460917e3 nodeName:}" failed. No retries permitted until 2026-07-09 17:17:54.83146581 +0000 UTC m=+1615.528638116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls") pod "maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" (UID: "8e856ec2-c462-4dad-a4ab-eb69460917e3") : secret "maas-api-e2e-mt-add69b2a-serving-cert" not found
Jul 09 17:17:54.331877 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.331858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e856ec2-c462-4dad-a4ab-eb69460917e3-openshift-service-ca\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.342174 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.342153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw92\" (UniqueName: \"kubernetes.io/projected/8e856ec2-c462-4dad-a4ab-eb69460917e3-kube-api-access-gvw92\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.460606 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.460577    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:17:54.835952 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.835922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.838375 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.838351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls\") pod \"maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:54.981340 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:54.981303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:55.109135 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:55.109111    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"]
Jul 09 17:17:55.111636 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:17:55.111612    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e856ec2_c462_4dad_a4ab_eb69460917e3.slice/crio-b3670bd25443f6138fba0f3d3492f39ed1488d6650267f441d2aec69764d1e22 WatchSource:0}: Error finding container b3670bd25443f6138fba0f3d3492f39ed1488d6650267f441d2aec69764d1e22: Status 404 returned error can't find the container with id b3670bd25443f6138fba0f3d3492f39ed1488d6650267f441d2aec69764d1e22
Jul 09 17:17:55.949578 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:55.949540    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" event={"ID":"8e856ec2-c462-4dad-a4ab-eb69460917e3","Type":"ContainerStarted","Data":"0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3"}
Jul 09 17:17:55.949578 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:55.949586    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" event={"ID":"8e856ec2-c462-4dad-a4ab-eb69460917e3","Type":"ContainerStarted","Data":"b3670bd25443f6138fba0f3d3492f39ed1488d6650267f441d2aec69764d1e22"}
Jul 09 17:17:55.949847 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:55.949677    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:17:55.965781 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:17:55.965732    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" podStartSLOduration=1.518673043 podStartE2EDuration="1.965717351s" totalImagesPullingTime="447.044308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:17:55.965171894 +0000 UTC m=+1616.662344220" watchObservedRunningTime="2026-07-09 17:17:55.965717351 +0000 UTC m=+1616.662889676"
Jul 09 17:18:01.960210 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:01.960179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:18:04.523013 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:04.522974    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:18:06.360500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.360471    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"]
Jul 09 17:18:06.361499 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.360670    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" podUID="8e856ec2-c462-4dad-a4ab-eb69460917e3" containerName="maas-api" containerID="cri-o://0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3" gracePeriod=30
Jul 09 17:18:06.608012 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.607986    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:18:06.726824 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.726727    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls\") pod \"8e856ec2-c462-4dad-a4ab-eb69460917e3\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") "
Jul 09 17:18:06.726972 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.726847    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e856ec2-c462-4dad-a4ab-eb69460917e3-openshift-service-ca\") pod \"8e856ec2-c462-4dad-a4ab-eb69460917e3\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") "
Jul 09 17:18:06.726972 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.726876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvw92\" (UniqueName: \"kubernetes.io/projected/8e856ec2-c462-4dad-a4ab-eb69460917e3-kube-api-access-gvw92\") pod \"8e856ec2-c462-4dad-a4ab-eb69460917e3\" (UID: \"8e856ec2-c462-4dad-a4ab-eb69460917e3\") "
Jul 09 17:18:06.727196 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.727169    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e856ec2-c462-4dad-a4ab-eb69460917e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e856ec2-c462-4dad-a4ab-eb69460917e3" (UID: "8e856ec2-c462-4dad-a4ab-eb69460917e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:18:06.728934 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.728901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e856ec2-c462-4dad-a4ab-eb69460917e3" (UID: "8e856ec2-c462-4dad-a4ab-eb69460917e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:18:06.729050 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.729010    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e856ec2-c462-4dad-a4ab-eb69460917e3-kube-api-access-gvw92" (OuterVolumeSpecName: "kube-api-access-gvw92") pod "8e856ec2-c462-4dad-a4ab-eb69460917e3" (UID: "8e856ec2-c462-4dad-a4ab-eb69460917e3"). InnerVolumeSpecName "kube-api-access-gvw92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:18:06.827477 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.827444    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e856ec2-c462-4dad-a4ab-eb69460917e3-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:18:06.827477 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.827474    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvw92\" (UniqueName: \"kubernetes.io/projected/8e856ec2-c462-4dad-a4ab-eb69460917e3-kube-api-access-gvw92\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:18:06.827477 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:06.827483    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e856ec2-c462-4dad-a4ab-eb69460917e3-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:18:07.002465 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.002397    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e856ec2-c462-4dad-a4ab-eb69460917e3" containerID="0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3" exitCode=0
Jul 09 17:18:07.002465 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.002432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" event={"ID":"8e856ec2-c462-4dad-a4ab-eb69460917e3","Type":"ContainerDied","Data":"0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3"}
Jul 09 17:18:07.002465 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.002436    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"
Jul 09 17:18:07.002465 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.002468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s" event={"ID":"8e856ec2-c462-4dad-a4ab-eb69460917e3","Type":"ContainerDied","Data":"b3670bd25443f6138fba0f3d3492f39ed1488d6650267f441d2aec69764d1e22"}
Jul 09 17:18:07.002729 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.002489    2567 scope.go:117] "RemoveContainer" containerID="0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3"
Jul 09 17:18:07.011067 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.011054    2567 scope.go:117] "RemoveContainer" containerID="0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3"
Jul 09 17:18:07.011299 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:18:07.011283    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3\": container with ID starting with 0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3 not found: ID does not exist" containerID="0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3"
Jul 09 17:18:07.011339 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.011307    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3"} err="failed to get container status \"0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3\": rpc error: code = NotFound desc = could not find container \"0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3\": container with ID starting with 0993f25c8765ce4a6803c89f17ce04bedfdabe856db1e862d5694630f57c35d3 not found: ID does not exist"
Jul 09 17:18:07.022624 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.022600    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"]
Jul 09 17:18:07.026913 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.026893    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-add69b2a-5597d75d4f-4ct2s"]
Jul 09 17:18:07.871615 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:07.871581    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e856ec2-c462-4dad-a4ab-eb69460917e3" path="/var/lib/kubelet/pods/8e856ec2-c462-4dad-a4ab-eb69460917e3/volumes"
Jul 09 17:18:14.762361 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:14.762320    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:18:35.187375 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.187332    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"]
Jul 09 17:18:35.187783 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.187766    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e856ec2-c462-4dad-a4ab-eb69460917e3" containerName="maas-api"
Jul 09 17:18:35.187783 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.187784    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e856ec2-c462-4dad-a4ab-eb69460917e3" containerName="maas-api"
Jul 09 17:18:35.187894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.187868    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e856ec2-c462-4dad-a4ab-eb69460917e3" containerName="maas-api"
Jul 09 17:18:35.190709 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.190687    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.193203 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.193180    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8384063d-serving-cert\""
Jul 09 17:18:35.193502 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.193481    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-whstr\""
Jul 09 17:18:35.194509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.194492    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:18:35.194509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.194503    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:18:35.201566 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.201544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"]
Jul 09 17:18:35.257248 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.257219    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.257397 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.257280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5bsf\" (UniqueName: \"kubernetes.io/projected/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-kube-api-access-h5bsf\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.257397 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.257301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-openshift-service-ca\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.358374 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.358342    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5bsf\" (UniqueName: \"kubernetes.io/projected/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-kube-api-access-h5bsf\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.358510 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.358378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-openshift-service-ca\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.358510 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.358449    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.358629 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:18:35.358557    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-serving-cert: secret "maas-api-e2e-mt-8384063d-serving-cert" not found
Jul 09 17:18:35.358683 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:18:35.358637    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls podName:efdfb21e-6aef-4a53-8a22-8b19cd7a47b9 nodeName:}" failed. No retries permitted until 2026-07-09 17:18:35.85861562 +0000 UTC m=+1656.555787942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls") pod "maas-api-e2e-mt-8384063d-669549cd5d-d7drr" (UID: "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9") : secret "maas-api-e2e-mt-8384063d-serving-cert" not found
Jul 09 17:18:35.359237 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.359216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-openshift-service-ca\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.369364 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.369337    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5bsf\" (UniqueName: \"kubernetes.io/projected/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-kube-api-access-h5bsf\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.863321 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.863284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:35.865869 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:35.865847    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls\") pod \"maas-api-e2e-mt-8384063d-669549cd5d-d7drr\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:36.102018 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:36.101989    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:36.242981 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:36.242911    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"]
Jul 09 17:18:36.245829 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:18:36.245787    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdfb21e_6aef_4a53_8a22_8b19cd7a47b9.slice/crio-dcaed3a8d83410c025a937c008edaec506531d203dcc1eb55a04869a8c320658 WatchSource:0}: Error finding container dcaed3a8d83410c025a937c008edaec506531d203dcc1eb55a04869a8c320658: Status 404 returned error can't find the container with id dcaed3a8d83410c025a937c008edaec506531d203dcc1eb55a04869a8c320658
Jul 09 17:18:37.136187 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:37.136153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr" event={"ID":"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9","Type":"ContainerStarted","Data":"06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e"}
Jul 09 17:18:37.136344 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:37.136204    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr" event={"ID":"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9","Type":"ContainerStarted","Data":"dcaed3a8d83410c025a937c008edaec506531d203dcc1eb55a04869a8c320658"}
Jul 09 17:18:37.136344 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:37.136234    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:37.153316 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:37.153270    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr" podStartSLOduration=1.6326669809999999 podStartE2EDuration="2.153256457s" totalImagesPullingTime="520.589476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:18:37.15211014 +0000 UTC m=+1657.849282468" watchObservedRunningTime="2026-07-09 17:18:37.153256457 +0000 UTC m=+1657.850428785"
Jul 09 17:18:41.561913 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:41.561868    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:18:43.146051 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:43.146024    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:18:44.243430 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.243393    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"]
Jul 09 17:18:44.247198 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.247176    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.249771 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.249746    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8384063d\"/\"openshift-service-ca.crt\""
Jul 09 17:18:44.249876 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.249749    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8384063d\"/\"default-dockercfg-jh7lz\""
Jul 09 17:18:44.250847 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.250827    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8384063d\"/\"e2e-lifecycle-model-8384063d-kserve-self-signed-certs\""
Jul 09 17:18:44.250946 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.250826    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8384063d\"/\"kube-root-ca.crt\""
Jul 09 17:18:44.256444 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.256409    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"]
Jul 09 17:18:44.333740 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.333715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-tmp-dir\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.333875 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.333747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-model-cache\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.333875 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.333775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjcv\" (UniqueName: \"kubernetes.io/projected/a5039e29-573a-4fee-8341-32ae914b54bd-kube-api-access-nnjcv\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.333875 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.333867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-home\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.333982 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.333934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-dshm\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.333982 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.333953    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5039e29-573a-4fee-8341-32ae914b54bd-tls-certs\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435040 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-home\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435158 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-dshm\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435158 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5039e29-573a-4fee-8341-32ae914b54bd-tls-certs\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435158 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-tmp-dir\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435299 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-model-cache\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435299 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435198    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjcv\" (UniqueName: \"kubernetes.io/projected/a5039e29-573a-4fee-8341-32ae914b54bd-kube-api-access-nnjcv\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435422 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435399    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-home\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435536 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435512    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-model-cache\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.435592 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.435550    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-tmp-dir\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.437414 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.437393    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-dshm\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.437675 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.437657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5039e29-573a-4fee-8341-32ae914b54bd-tls-certs\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.442621 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.442600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjcv\" (UniqueName: \"kubernetes.io/projected/a5039e29-573a-4fee-8341-32ae914b54bd-kube-api-access-nnjcv\") pod \"e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") " pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.559296 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.559235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:44.692448 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:44.692420    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"]
Jul 09 17:18:44.694683 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:18:44.694654    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5039e29_573a_4fee_8341_32ae914b54bd.slice/crio-b42a0498117b3f6682a1548a9cc32689086f223bedc47608d00c3b7edbd2b2bb WatchSource:0}: Error finding container b42a0498117b3f6682a1548a9cc32689086f223bedc47608d00c3b7edbd2b2bb: Status 404 returned error can't find the container with id b42a0498117b3f6682a1548a9cc32689086f223bedc47608d00c3b7edbd2b2bb
Jul 09 17:18:45.172229 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:45.172196    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt" event={"ID":"a5039e29-573a-4fee-8341-32ae914b54bd","Type":"ContainerStarted","Data":"f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d"}
Jul 09 17:18:45.172363 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:45.172235    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt" event={"ID":"a5039e29-573a-4fee-8341-32ae914b54bd","Type":"ContainerStarted","Data":"b42a0498117b3f6682a1548a9cc32689086f223bedc47608d00c3b7edbd2b2bb"}
Jul 09 17:18:45.188481 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:45.188432    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt" podStartSLOduration=1.188417775 podStartE2EDuration="1.188417775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:18:45.187524597 +0000 UTC m=+1665.884696926" watchObservedRunningTime="2026-07-09 17:18:45.188417775 +0000 UTC m=+1665.885590080"
Jul 09 17:18:53.861169 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:53.861134    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:18:54.559980 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:54.559947    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:54.560237 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:54.560207    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:54.572452 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:54.572435    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:18:55.225913 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:18:55.225875    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:19:09.962662 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:09.962628    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:19:12.549993 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.549955    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"]
Jul 09 17:19:12.550463 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.550248    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr" podUID="efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" containerName="maas-api" containerID="cri-o://06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e" gracePeriod=30
Jul 09 17:19:12.790123 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.790098    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:19:12.877781 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.877753    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls\") pod \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") "
Jul 09 17:19:12.877950 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.877814    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5bsf\" (UniqueName: \"kubernetes.io/projected/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-kube-api-access-h5bsf\") pod \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") "
Jul 09 17:19:12.877950 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.877834    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-openshift-service-ca\") pod \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\" (UID: \"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9\") "
Jul 09 17:19:12.878247 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.878222    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" (UID: "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:19:12.879925 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.879902    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" (UID: "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:19:12.880038 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.880020    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-kube-api-access-h5bsf" (OuterVolumeSpecName: "kube-api-access-h5bsf") pod "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" (UID: "efdfb21e-6aef-4a53-8a22-8b19cd7a47b9"). InnerVolumeSpecName "kube-api-access-h5bsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:19:12.978376 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.978341    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5bsf\" (UniqueName: \"kubernetes.io/projected/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-kube-api-access-h5bsf\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:12.978376 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.978368    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:12.978376 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:12.978380    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:13.301545 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.301453    2567 generic.go:356] "Generic (PLEG): container finished" podID="efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" containerID="06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e" exitCode=0
Jul 09 17:19:13.301545 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.301501    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"
Jul 09 17:19:13.301545 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.301533    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr" event={"ID":"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9","Type":"ContainerDied","Data":"06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e"}
Jul 09 17:19:13.301749 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.301616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr" event={"ID":"efdfb21e-6aef-4a53-8a22-8b19cd7a47b9","Type":"ContainerDied","Data":"dcaed3a8d83410c025a937c008edaec506531d203dcc1eb55a04869a8c320658"}
Jul 09 17:19:13.301749 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.301647    2567 scope.go:117] "RemoveContainer" containerID="06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e"
Jul 09 17:19:13.310462 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.310444    2567 scope.go:117] "RemoveContainer" containerID="06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e"
Jul 09 17:19:13.310702 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:19:13.310685    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e\": container with ID starting with 06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e not found: ID does not exist" containerID="06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e"
Jul 09 17:19:13.310748 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.310714    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e"} err="failed to get container status \"06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e\": rpc error: code = NotFound desc = could not find container \"06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e\": container with ID starting with 06b9cbec272186760bd669a10c0b0ca766724b0c679da45305cfe816ce24de6e not found: ID does not exist"
Jul 09 17:19:13.321856 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.321835    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"]
Jul 09 17:19:13.325496 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.325477    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8384063d-669549cd5d-d7drr"]
Jul 09 17:19:13.872601 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:13.872566    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" path="/var/lib/kubelet/pods/efdfb21e-6aef-4a53-8a22-8b19cd7a47b9/volumes"
Jul 09 17:19:22.753146 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:22.753068    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:19:23.317890 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:23.317852    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"]
Jul 09 17:19:38.339652 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.339581    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt" podUID="a5039e29-573a-4fee-8341-32ae914b54bd" containerName="main" containerID="cri-o://f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d" gracePeriod=45
Jul 09 17:19:38.591659 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.591606    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:19:38.690018 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.689993    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-model-cache\") pod \"a5039e29-573a-4fee-8341-32ae914b54bd\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") "
Jul 09 17:19:38.690157 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-home\") pod \"a5039e29-573a-4fee-8341-32ae914b54bd\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") "
Jul 09 17:19:38.690157 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690045    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-dshm\") pod \"a5039e29-573a-4fee-8341-32ae914b54bd\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") "
Jul 09 17:19:38.690157 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690074    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5039e29-573a-4fee-8341-32ae914b54bd-tls-certs\") pod \"a5039e29-573a-4fee-8341-32ae914b54bd\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") "
Jul 09 17:19:38.690157 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690150    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-tmp-dir\") pod \"a5039e29-573a-4fee-8341-32ae914b54bd\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") "
Jul 09 17:19:38.690395 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690192    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnjcv\" (UniqueName: \"kubernetes.io/projected/a5039e29-573a-4fee-8341-32ae914b54bd-kube-api-access-nnjcv\") pod \"a5039e29-573a-4fee-8341-32ae914b54bd\" (UID: \"a5039e29-573a-4fee-8341-32ae914b54bd\") "
Jul 09 17:19:38.690395 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690281    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-home" (OuterVolumeSpecName: "home") pod "a5039e29-573a-4fee-8341-32ae914b54bd" (UID: "a5039e29-573a-4fee-8341-32ae914b54bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:19:38.690395 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690295    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5039e29-573a-4fee-8341-32ae914b54bd" (UID: "a5039e29-573a-4fee-8341-32ae914b54bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:19:38.690543 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690479    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:38.690543 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690483    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5039e29-573a-4fee-8341-32ae914b54bd" (UID: "a5039e29-573a-4fee-8341-32ae914b54bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:19:38.690543 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.690498    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:38.692400 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.692378    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-dshm" (OuterVolumeSpecName: "dshm") pod "a5039e29-573a-4fee-8341-32ae914b54bd" (UID: "a5039e29-573a-4fee-8341-32ae914b54bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:19:38.692400 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.692383    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5039e29-573a-4fee-8341-32ae914b54bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5039e29-573a-4fee-8341-32ae914b54bd" (UID: "a5039e29-573a-4fee-8341-32ae914b54bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:19:38.692529 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.692397    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5039e29-573a-4fee-8341-32ae914b54bd-kube-api-access-nnjcv" (OuterVolumeSpecName: "kube-api-access-nnjcv") pod "a5039e29-573a-4fee-8341-32ae914b54bd" (UID: "a5039e29-573a-4fee-8341-32ae914b54bd"). InnerVolumeSpecName "kube-api-access-nnjcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:19:38.791133 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.791105    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:38.791133 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.791129    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5039e29-573a-4fee-8341-32ae914b54bd-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:38.791267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.791139    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5039e29-573a-4fee-8341-32ae914b54bd-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:38.791267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:38.791148    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjcv\" (UniqueName: \"kubernetes.io/projected/a5039e29-573a-4fee-8341-32ae914b54bd-kube-api-access-nnjcv\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:19:39.427840 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.427778    2567 generic.go:356] "Generic (PLEG): container finished" podID="a5039e29-573a-4fee-8341-32ae914b54bd" containerID="f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d" exitCode=0
Jul 09 17:19:39.427840 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.427828    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt" event={"ID":"a5039e29-573a-4fee-8341-32ae914b54bd","Type":"ContainerDied","Data":"f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d"}
Jul 09 17:19:39.428344 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.427861    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"
Jul 09 17:19:39.428344 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.427884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt" event={"ID":"a5039e29-573a-4fee-8341-32ae914b54bd","Type":"ContainerDied","Data":"b42a0498117b3f6682a1548a9cc32689086f223bedc47608d00c3b7edbd2b2bb"}
Jul 09 17:19:39.428344 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.427903    2567 scope.go:117] "RemoveContainer" containerID="f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d"
Jul 09 17:19:39.438628 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.438607    2567 scope.go:117] "RemoveContainer" containerID="f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d"
Jul 09 17:19:39.438997 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:19:39.438977    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d\": container with ID starting with f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d not found: ID does not exist" containerID="f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d"
Jul 09 17:19:39.439041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.439007    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d"} err="failed to get container status \"f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d\": rpc error: code = NotFound desc = could not find container \"f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d\": container with ID starting with f8ff6fa1bee76327923756f3d4ea162e47def220fbf1075819dbf655e0d70b0d not found: ID does not exist"
Jul 09 17:19:39.451836 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.451809    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"]
Jul 09 17:19:39.456506 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.456481    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8384063d/e2e-lifecycle-model-8384063d-kserve-79f48879ff-h26dt"]
Jul 09 17:19:39.873484 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:19:39.873456    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5039e29-573a-4fee-8341-32ae914b54bd" path="/var/lib/kubelet/pods/a5039e29-573a-4fee-8341-32ae914b54bd/volumes"
Jul 09 17:20:03.552586 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:20:03.552556    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:20:24.571568 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:20:24.571530    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:20:39.062978 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:20:39.062943    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:21:18.013824 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.013766    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"]
Jul 09 17:21:18.014316 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.014291    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5039e29-573a-4fee-8341-32ae914b54bd" containerName="main"
Jul 09 17:21:18.014316 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.014318    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5039e29-573a-4fee-8341-32ae914b54bd" containerName="main"
Jul 09 17:21:18.014473 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.014346    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" containerName="maas-api"
Jul 09 17:21:18.014473 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.014356    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" containerName="maas-api"
Jul 09 17:21:18.014473 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.014451    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efdfb21e-6aef-4a53-8a22-8b19cd7a47b9" containerName="maas-api"
Jul 09 17:21:18.014473 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.014466    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5039e29-573a-4fee-8341-32ae914b54bd" containerName="main"
Jul 09 17:21:18.017715 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.017697    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.020474 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.020451    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-88cd44-serving-cert\""
Jul 09 17:21:18.021698 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.021680    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:21:18.021848 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.021685    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-whstr\""
Jul 09 17:21:18.021848 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.021733    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:21:18.030220 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.030197    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"]
Jul 09 17:21:18.049506 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.049475    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.049629 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.049532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcc6\" (UniqueName: \"kubernetes.io/projected/30c6d32d-50dd-48d3-9010-9c13d26d28de-kube-api-access-vgcc6\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.049629 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.049605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6d32d-50dd-48d3-9010-9c13d26d28de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.150836 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.150777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6d32d-50dd-48d3-9010-9c13d26d28de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.151029 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.150869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.151029 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.150919    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcc6\" (UniqueName: \"kubernetes.io/projected/30c6d32d-50dd-48d3-9010-9c13d26d28de-kube-api-access-vgcc6\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.151161 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:21:18.151023    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-serving-cert: secret "maas-api-e2e-inf-a-88cd44-serving-cert" not found
Jul 09 17:21:18.151161 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:21:18.151112    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls podName:30c6d32d-50dd-48d3-9010-9c13d26d28de nodeName:}" failed. No retries permitted until 2026-07-09 17:21:18.651089869 +0000 UTC m=+1819.348262177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls") pod "maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" (UID: "30c6d32d-50dd-48d3-9010-9c13d26d28de") : secret "maas-api-e2e-inf-a-88cd44-serving-cert" not found
Jul 09 17:21:18.151396 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.151376    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6d32d-50dd-48d3-9010-9c13d26d28de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.161466 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.161439    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcc6\" (UniqueName: \"kubernetes.io/projected/30c6d32d-50dd-48d3-9010-9c13d26d28de-kube-api-access-vgcc6\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.654384 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.654353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.656943 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.656913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls\") pod \"maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:18.927865 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:18.927762    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:19.056596 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:19.056563    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"]
Jul 09 17:21:19.059105 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:21:19.059069    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c6d32d_50dd_48d3_9010_9c13d26d28de.slice/crio-44c2673d97ddef77bc69cfce976fa880463adc379e8ee01b282976ae363200d6 WatchSource:0}: Error finding container 44c2673d97ddef77bc69cfce976fa880463adc379e8ee01b282976ae363200d6: Status 404 returned error can't find the container with id 44c2673d97ddef77bc69cfce976fa880463adc379e8ee01b282976ae363200d6
Jul 09 17:21:19.060523 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:19.060505    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:21:19.887563 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:19.887526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" event={"ID":"30c6d32d-50dd-48d3-9010-9c13d26d28de","Type":"ContainerStarted","Data":"e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939"}
Jul 09 17:21:19.887744 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:19.887569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" event={"ID":"30c6d32d-50dd-48d3-9010-9c13d26d28de","Type":"ContainerStarted","Data":"44c2673d97ddef77bc69cfce976fa880463adc379e8ee01b282976ae363200d6"}
Jul 09 17:21:19.887744 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:19.887619    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:19.908469 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:19.908423    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" podStartSLOduration=2.486987558 podStartE2EDuration="2.90840914s" totalImagesPullingTime="421.421582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:21:19.907349406 +0000 UTC m=+1820.604521731" watchObservedRunningTime="2026-07-09 17:21:19.90840914 +0000 UTC m=+1820.605581467"
Jul 09 17:21:25.897049 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:25.897020    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:21:25.981904 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:25.981864    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:21:26.165272 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.165191    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"]
Jul 09 17:21:26.168938 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.168916    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.171229 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.171210    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-33cf76-serving-cert\""
Jul 09 17:21:26.177742 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.177717    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"]
Jul 09 17:21:26.217341 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.217311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4345c-88fb-4e9a-a296-b209fdd7668a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.217512 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.217361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59w8c\" (UniqueName: \"kubernetes.io/projected/efd4345c-88fb-4e9a-a296-b209fdd7668a-kube-api-access-59w8c\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.217512 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.217398    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.318435 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.318402    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4345c-88fb-4e9a-a296-b209fdd7668a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.318629 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.318460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59w8c\" (UniqueName: \"kubernetes.io/projected/efd4345c-88fb-4e9a-a296-b209fdd7668a-kube-api-access-59w8c\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.318629 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.318503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.318751 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:21:26.318634    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-serving-cert: secret "maas-api-e2e-inf-b-33cf76-serving-cert" not found
Jul 09 17:21:26.318751 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:21:26.318718    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls podName:efd4345c-88fb-4e9a-a296-b209fdd7668a nodeName:}" failed. No retries permitted until 2026-07-09 17:21:26.818695587 +0000 UTC m=+1827.515867911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls") pod "maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" (UID: "efd4345c-88fb-4e9a-a296-b209fdd7668a") : secret "maas-api-e2e-inf-b-33cf76-serving-cert" not found
Jul 09 17:21:26.319196 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.319175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4345c-88fb-4e9a-a296-b209fdd7668a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.327203 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.327176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59w8c\" (UniqueName: \"kubernetes.io/projected/efd4345c-88fb-4e9a-a296-b209fdd7668a-kube-api-access-59w8c\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.824166 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.824123    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:26.826637 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:26.826612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls\") pod \"maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:27.081287 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:27.081192    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:27.214262 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:27.214235    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"]
Jul 09 17:21:27.216616 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:21:27.216590    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd4345c_88fb_4e9a_a296_b209fdd7668a.slice/crio-68d57bde3ae96b1a6f593d0bf3b1715559527d8887066e832dae33889e4d86f3 WatchSource:0}: Error finding container 68d57bde3ae96b1a6f593d0bf3b1715559527d8887066e832dae33889e4d86f3: Status 404 returned error can't find the container with id 68d57bde3ae96b1a6f593d0bf3b1715559527d8887066e832dae33889e4d86f3
Jul 09 17:21:27.922770 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:27.922725    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" event={"ID":"efd4345c-88fb-4e9a-a296-b209fdd7668a","Type":"ContainerStarted","Data":"3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c"}
Jul 09 17:21:27.922770 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:27.922776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" event={"ID":"efd4345c-88fb-4e9a-a296-b209fdd7668a","Type":"ContainerStarted","Data":"68d57bde3ae96b1a6f593d0bf3b1715559527d8887066e832dae33889e4d86f3"}
Jul 09 17:21:27.923037 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:27.922875    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:27.939221 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:27.939157    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" podStartSLOduration=1.578652989 podStartE2EDuration="1.939143136s" totalImagesPullingTime="360.490147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:21:27.938817192 +0000 UTC m=+1828.635989517" watchObservedRunningTime="2026-07-09 17:21:27.939143136 +0000 UTC m=+1828.636315496"
Jul 09 17:21:33.933722 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:33.933686    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:21:34.434611 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.434569    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"]
Jul 09 17:21:34.437919 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.437898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.440370 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.440341    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-88cd44\"/\"kube-root-ca.crt\""
Jul 09 17:21:34.440476 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.440413    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-88cd44\"/\"openshift-service-ca.crt\""
Jul 09 17:21:34.441808 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.441775    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-88cd44\"/\"default-dockercfg-9zdkm\""
Jul 09 17:21:34.441887 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.441826    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-88cd44\"/\"test-model-88cd44-kserve-self-signed-certs\""
Jul 09 17:21:34.445844 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.445826    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"]
Jul 09 17:21:34.484601 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.484574    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-dshm\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.484711 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.484608    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-model-cache\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.484711 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.484643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-tmp-dir\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.484882 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.484739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-home\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.484882 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.484765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkcw\" (UniqueName: \"kubernetes.io/projected/eb054748-2013-4748-a225-55929353d4ca-kube-api-access-pfkcw\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.484882 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.484785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb054748-2013-4748-a225-55929353d4ca-tls-certs\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.585958 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.585933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-home\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586098 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.585969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkcw\" (UniqueName: \"kubernetes.io/projected/eb054748-2013-4748-a225-55929353d4ca-kube-api-access-pfkcw\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586098 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.585989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb054748-2013-4748-a225-55929353d4ca-tls-certs\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586098 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.586060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-dshm\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586098 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.586084    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-model-cache\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586314 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.586119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-tmp-dir\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586376 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.586359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-home\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.586480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-tmp-dir\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.586582 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.586511    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-model-cache\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.588369 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.588344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-dshm\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.588628 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.588607    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb054748-2013-4748-a225-55929353d4ca-tls-certs\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.594436 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.594403    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkcw\" (UniqueName: \"kubernetes.io/projected/eb054748-2013-4748-a225-55929353d4ca-kube-api-access-pfkcw\") pod \"test-model-88cd44-kserve-6478ffc8c5-r546q\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") " pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.747886 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.747784    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:34.882974 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.882824    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"]
Jul 09 17:21:34.885365 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:21:34.885341    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb054748_2013_4748_a225_55929353d4ca.slice/crio-2f5ca9bd5e510234aa4287a00ec5143b35f5a17c764cd7d4f902f7b32fd5eebf WatchSource:0}: Error finding container 2f5ca9bd5e510234aa4287a00ec5143b35f5a17c764cd7d4f902f7b32fd5eebf: Status 404 returned error can't find the container with id 2f5ca9bd5e510234aa4287a00ec5143b35f5a17c764cd7d4f902f7b32fd5eebf
Jul 09 17:21:34.969859 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.969820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q" event={"ID":"eb054748-2013-4748-a225-55929353d4ca","Type":"ContainerStarted","Data":"db3b72f475642bd79d0417b1477f21fca5cf60ecea913f55d4fc9c6cd9b4873c"}
Jul 09 17:21:34.970285 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.969871    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q" event={"ID":"eb054748-2013-4748-a225-55929353d4ca","Type":"ContainerStarted","Data":"2f5ca9bd5e510234aa4287a00ec5143b35f5a17c764cd7d4f902f7b32fd5eebf"}
Jul 09 17:21:34.988256 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:34.988211    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q" podStartSLOduration=0.988196671 podStartE2EDuration="988.196671ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:21:34.987740511 +0000 UTC m=+1835.684912839" watchObservedRunningTime="2026-07-09 17:21:34.988196671 +0000 UTC m=+1835.685368997"
Jul 09 17:21:38.162315 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:38.162281    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:21:44.748728 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:44.748685    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:44.751152 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:44.748751    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:44.761319 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:44.761297    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:21:45.026358 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:21:45.026277    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:22:04.651180 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.651148    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"]
Jul 09 17:22:04.661009 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.660970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.663881 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.663782    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"]
Jul 09 17:22:04.665022 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.664999    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-33cf76\"/\"kube-root-ca.crt\""
Jul 09 17:22:04.665172 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.665000    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-33cf76\"/\"test-model-33cf76-kserve-self-signed-certs\""
Jul 09 17:22:04.665314 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.665282    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-33cf76\"/\"openshift-service-ca.crt\""
Jul 09 17:22:04.665518 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.665320    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-33cf76\"/\"default-dockercfg-rhwcv\""
Jul 09 17:22:04.750351 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.750315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tmp-dir\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.750351 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.750347    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khknz\" (UniqueName: \"kubernetes.io/projected/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-kube-api-access-khknz\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.750553 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.750401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-model-cache\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.750553 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.750420    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-dshm\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.750553 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.750454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-home\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.750553 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.750480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tls-certs\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.851921 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.851896    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tls-certs\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852102 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.851951    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tmp-dir\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852102 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.851979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khknz\" (UniqueName: \"kubernetes.io/projected/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-kube-api-access-khknz\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852102 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.852051    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-model-cache\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852102 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.852079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-dshm\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852308 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.852111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-home\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852370 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.852349    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tmp-dir\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852415 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.852374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-model-cache\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.852465 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.852450    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-home\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.854467 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.854447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-dshm\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.854561 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.854515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tls-certs\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.860741 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.860721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khknz\" (UniqueName: \"kubernetes.io/projected/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-kube-api-access-khknz\") pod \"test-model-33cf76-kserve-f947c879c-r5plc\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") " pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:04.971781 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:04.971705    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:05.108234 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:05.108213    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"]
Jul 09 17:22:05.110177 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:22:05.110144    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68c1a08c_16d8_4c74_a2e0_1bfbf61365bf.slice/crio-3a40f0f213f3918424382f00eb1e8fb63accf73e9cf2f8b23e76d37470041530 WatchSource:0}: Error finding container 3a40f0f213f3918424382f00eb1e8fb63accf73e9cf2f8b23e76d37470041530: Status 404 returned error can't find the container with id 3a40f0f213f3918424382f00eb1e8fb63accf73e9cf2f8b23e76d37470041530
Jul 09 17:22:06.112768 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:06.112724    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" event={"ID":"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf","Type":"ContainerStarted","Data":"b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c"}
Jul 09 17:22:06.112768 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:06.112764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" event={"ID":"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf","Type":"ContainerStarted","Data":"3a40f0f213f3918424382f00eb1e8fb63accf73e9cf2f8b23e76d37470041530"}
Jul 09 17:22:06.130099 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:06.130047    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" podStartSLOduration=2.130028869 podStartE2EDuration="2.130028869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:22:06.129492487 +0000 UTC m=+1866.826664815" watchObservedRunningTime="2026-07-09 17:22:06.130028869 +0000 UTC m=+1866.827201196"
Jul 09 17:22:09.062424 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:09.062391    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:22:14.972865 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:14.972791    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:14.973276 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:14.972886    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:14.985340 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:14.985312    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:15.164642 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:15.164604    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:22.563871 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:22.563775    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:22:29.061090 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:29.061050    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"]
Jul 09 17:22:29.966050 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:29.966013    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"]
Jul 09 17:22:30.882578 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:30.882541    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"]
Jul 09 17:22:30.883017 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:30.882777    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" podUID="30c6d32d-50dd-48d3-9010-9c13d26d28de" containerName="maas-api" containerID="cri-o://e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939" gracePeriod=30
Jul 09 17:22:30.892344 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:30.892301    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" podUID="30c6d32d-50dd-48d3-9010-9c13d26d28de" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.33:8443/health\": dial tcp 10.133.0.33:8443: connect: connection refused"
Jul 09 17:22:31.132221 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.132195    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:22:31.225610 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.225516    2567 generic.go:356] "Generic (PLEG): container finished" podID="30c6d32d-50dd-48d3-9010-9c13d26d28de" containerID="e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939" exitCode=0
Jul 09 17:22:31.225610 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.225557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" event={"ID":"30c6d32d-50dd-48d3-9010-9c13d26d28de","Type":"ContainerDied","Data":"e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939"}
Jul 09 17:22:31.225610 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.225572    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"
Jul 09 17:22:31.225610 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.225597    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf" event={"ID":"30c6d32d-50dd-48d3-9010-9c13d26d28de","Type":"ContainerDied","Data":"44c2673d97ddef77bc69cfce976fa880463adc379e8ee01b282976ae363200d6"}
Jul 09 17:22:31.226008 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.225618    2567 scope.go:117] "RemoveContainer" containerID="e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939"
Jul 09 17:22:31.234544 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.234524    2567 scope.go:117] "RemoveContainer" containerID="e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939"
Jul 09 17:22:31.234823 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:22:31.234784    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939\": container with ID starting with e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939 not found: ID does not exist" containerID="e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939"
Jul 09 17:22:31.234886 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.234830    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939"} err="failed to get container status \"e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939\": rpc error: code = NotFound desc = could not find container \"e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939\": container with ID starting with e52ead4e676a88fdb23394e2d32144f62a469b8e260d401a50bd4d7f2b64c939 not found: ID does not exist"
Jul 09 17:22:31.291926 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.291889    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6d32d-50dd-48d3-9010-9c13d26d28de-openshift-service-ca\") pod \"30c6d32d-50dd-48d3-9010-9c13d26d28de\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") "
Jul 09 17:22:31.292089 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.291971    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgcc6\" (UniqueName: \"kubernetes.io/projected/30c6d32d-50dd-48d3-9010-9c13d26d28de-kube-api-access-vgcc6\") pod \"30c6d32d-50dd-48d3-9010-9c13d26d28de\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") "
Jul 09 17:22:31.292089 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.292008    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls\") pod \"30c6d32d-50dd-48d3-9010-9c13d26d28de\" (UID: \"30c6d32d-50dd-48d3-9010-9c13d26d28de\") "
Jul 09 17:22:31.292332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.292294    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c6d32d-50dd-48d3-9010-9c13d26d28de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30c6d32d-50dd-48d3-9010-9c13d26d28de" (UID: "30c6d32d-50dd-48d3-9010-9c13d26d28de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:22:31.294280 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.294257    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30c6d32d-50dd-48d3-9010-9c13d26d28de" (UID: "30c6d32d-50dd-48d3-9010-9c13d26d28de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:22:31.294552 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.294537    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c6d32d-50dd-48d3-9010-9c13d26d28de-kube-api-access-vgcc6" (OuterVolumeSpecName: "kube-api-access-vgcc6") pod "30c6d32d-50dd-48d3-9010-9c13d26d28de" (UID: "30c6d32d-50dd-48d3-9010-9c13d26d28de"). InnerVolumeSpecName "kube-api-access-vgcc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:22:31.392962 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.392918    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6d32d-50dd-48d3-9010-9c13d26d28de-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:31.392962 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.392960    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgcc6\" (UniqueName: \"kubernetes.io/projected/30c6d32d-50dd-48d3-9010-9c13d26d28de-kube-api-access-vgcc6\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:31.393136 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.392976    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30c6d32d-50dd-48d3-9010-9c13d26d28de-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:31.547703 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.547669    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"]
Jul 09 17:22:31.551692 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.551667    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-88cd44-c87c78dc9-4nqsf"]
Jul 09 17:22:31.871939 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:31.871910    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c6d32d-50dd-48d3-9010-9c13d26d28de" path="/var/lib/kubelet/pods/30c6d32d-50dd-48d3-9010-9c13d26d28de/volumes"
Jul 09 17:22:35.659453 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:35.659412    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:22:44.082472 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.082432    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q" podUID="eb054748-2013-4748-a225-55929353d4ca" containerName="main" containerID="cri-o://db3b72f475642bd79d0417b1477f21fca5cf60ecea913f55d4fc9c6cd9b4873c" gracePeriod=45
Jul 09 17:22:44.296085 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.296054    2567 generic.go:356] "Generic (PLEG): container finished" podID="eb054748-2013-4748-a225-55929353d4ca" containerID="db3b72f475642bd79d0417b1477f21fca5cf60ecea913f55d4fc9c6cd9b4873c" exitCode=0
Jul 09 17:22:44.296281 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.296107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q" event={"ID":"eb054748-2013-4748-a225-55929353d4ca","Type":"ContainerDied","Data":"db3b72f475642bd79d0417b1477f21fca5cf60ecea913f55d4fc9c6cd9b4873c"}
Jul 09 17:22:44.332719 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.332656    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:22:44.420887 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.420854    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb054748-2013-4748-a225-55929353d4ca-tls-certs\") pod \"eb054748-2013-4748-a225-55929353d4ca\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") "
Jul 09 17:22:44.420887 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.420902    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-tmp-dir\") pod \"eb054748-2013-4748-a225-55929353d4ca\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") "
Jul 09 17:22:44.421118 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.420955    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-dshm\") pod \"eb054748-2013-4748-a225-55929353d4ca\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") "
Jul 09 17:22:44.421118 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.420995    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfkcw\" (UniqueName: \"kubernetes.io/projected/eb054748-2013-4748-a225-55929353d4ca-kube-api-access-pfkcw\") pod \"eb054748-2013-4748-a225-55929353d4ca\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") "
Jul 09 17:22:44.421118 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-model-cache\") pod \"eb054748-2013-4748-a225-55929353d4ca\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") "
Jul 09 17:22:44.421118 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421071    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-home\") pod \"eb054748-2013-4748-a225-55929353d4ca\" (UID: \"eb054748-2013-4748-a225-55929353d4ca\") "
Jul 09 17:22:44.421327 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421240    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb054748-2013-4748-a225-55929353d4ca" (UID: "eb054748-2013-4748-a225-55929353d4ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:44.421386 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421344    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-home" (OuterVolumeSpecName: "home") pod "eb054748-2013-4748-a225-55929353d4ca" (UID: "eb054748-2013-4748-a225-55929353d4ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:44.421441 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421395    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb054748-2013-4748-a225-55929353d4ca" (UID: "eb054748-2013-4748-a225-55929353d4ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:44.421478 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421455    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:44.421478 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.421471    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:44.423348 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.423326    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-dshm" (OuterVolumeSpecName: "dshm") pod "eb054748-2013-4748-a225-55929353d4ca" (UID: "eb054748-2013-4748-a225-55929353d4ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:44.423617 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.423599    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb054748-2013-4748-a225-55929353d4ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb054748-2013-4748-a225-55929353d4ca" (UID: "eb054748-2013-4748-a225-55929353d4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:22:44.423617 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.423605    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb054748-2013-4748-a225-55929353d4ca-kube-api-access-pfkcw" (OuterVolumeSpecName: "kube-api-access-pfkcw") pod "eb054748-2013-4748-a225-55929353d4ca" (UID: "eb054748-2013-4748-a225-55929353d4ca"). InnerVolumeSpecName "kube-api-access-pfkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:22:44.522189 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.522143    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkcw\" (UniqueName: \"kubernetes.io/projected/eb054748-2013-4748-a225-55929353d4ca-kube-api-access-pfkcw\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:44.522189 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.522184    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:44.522189 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.522195    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb054748-2013-4748-a225-55929353d4ca-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:44.522189 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.522202    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb054748-2013-4748-a225-55929353d4ca-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:44.987238 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:44.987173    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" podUID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" containerName="main" containerID="cri-o://b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c" gracePeriod=45
Jul 09 17:22:45.235523 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.235499    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:45.301674 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.301598    2567 generic.go:356] "Generic (PLEG): container finished" podID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" containerID="b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c" exitCode=0
Jul 09 17:22:45.301674 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.301636    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"
Jul 09 17:22:45.301886 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.301679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" event={"ID":"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf","Type":"ContainerDied","Data":"b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c"}
Jul 09 17:22:45.301886 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.301717    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" event={"ID":"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf","Type":"ContainerDied","Data":"3a40f0f213f3918424382f00eb1e8fb63accf73e9cf2f8b23e76d37470041530"}
Jul 09 17:22:45.301886 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.301735    2567 scope.go:117] "RemoveContainer" containerID="b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c"
Jul 09 17:22:45.303051 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.303024    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"
Jul 09 17:22:45.303051 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.303041    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q" event={"ID":"eb054748-2013-4748-a225-55929353d4ca","Type":"ContainerDied","Data":"2f5ca9bd5e510234aa4287a00ec5143b35f5a17c764cd7d4f902f7b32fd5eebf"}
Jul 09 17:22:45.310650 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.310629    2567 scope.go:117] "RemoveContainer" containerID="b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c"
Jul 09 17:22:45.310946 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:22:45.310927    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c\": container with ID starting with b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c not found: ID does not exist" containerID="b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c"
Jul 09 17:22:45.311009 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.310954    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c"} err="failed to get container status \"b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c\": rpc error: code = NotFound desc = could not find container \"b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c\": container with ID starting with b059d8aad39f9eb89fb27549510d5dbf5c932e4b80cfc391db2e55dd9f0a116c not found: ID does not exist"
Jul 09 17:22:45.311009 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.310972    2567 scope.go:117] "RemoveContainer" containerID="db3b72f475642bd79d0417b1477f21fca5cf60ecea913f55d4fc9c6cd9b4873c"
Jul 09 17:22:45.323695 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.323676    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"]
Jul 09 17:22:45.327171 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.327149    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-88cd44/test-model-88cd44-kserve-6478ffc8c5-r546q"]
Jul 09 17:22:45.329583 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-dshm\") pod \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") "
Jul 09 17:22:45.329657 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329591    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-home\") pod \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") "
Jul 09 17:22:45.329657 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329645    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tls-certs\") pod \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") "
Jul 09 17:22:45.329740 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329690    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-model-cache\") pod \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") "
Jul 09 17:22:45.329861 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329844    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khknz\" (UniqueName: \"kubernetes.io/projected/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-kube-api-access-khknz\") pod \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") "
Jul 09 17:22:45.329931 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329910    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tmp-dir\") pod \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\" (UID: \"68c1a08c-16d8-4c74-a2e0-1bfbf61365bf\") "
Jul 09 17:22:45.329972 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329943    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" (UID: "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:45.330029 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.329961    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-home" (OuterVolumeSpecName: "home") pod "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" (UID: "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:45.330189 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.330169    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" (UID: "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:45.330298 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.330283    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:45.330352 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.330307    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:45.330352 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.330319    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:45.331846 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.331823    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-dshm" (OuterVolumeSpecName: "dshm") pod "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" (UID: "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:45.331936 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.331903    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-kube-api-access-khknz" (OuterVolumeSpecName: "kube-api-access-khknz") pod "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" (UID: "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf"). InnerVolumeSpecName "kube-api-access-khknz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:22:45.331994 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.331979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" (UID: "68c1a08c-16d8-4c74-a2e0-1bfbf61365bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:22:45.431563 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.431539    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khknz\" (UniqueName: \"kubernetes.io/projected/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-kube-api-access-khknz\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:45.431563 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.431559    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:45.431686 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.431569    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:22:45.622597 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.622571    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"]
Jul 09 17:22:45.626388 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.626367    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc"]
Jul 09 17:22:45.872707 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.872617    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" path="/var/lib/kubelet/pods/68c1a08c-16d8-4c74-a2e0-1bfbf61365bf/volumes"
Jul 09 17:22:45.873108 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:45.873086    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb054748-2013-4748-a225-55929353d4ca" path="/var/lib/kubelet/pods/eb054748-2013-4748-a225-55929353d4ca/volumes"
Jul 09 17:22:50.154899 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:22:50.154861    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-33cf76/test-model-33cf76-kserve-f947c879c-r5plc" podUID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Jul 09 17:23:14.066284 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.066239    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"]
Jul 09 17:23:14.066717 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.066479    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" podUID="efd4345c-88fb-4e9a-a296-b209fdd7668a" containerName="maas-api" containerID="cri-o://3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c" gracePeriod=30
Jul 09 17:23:14.310778 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.310754    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:23:14.438050 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.438016    2567 generic.go:356] "Generic (PLEG): container finished" podID="efd4345c-88fb-4e9a-a296-b209fdd7668a" containerID="3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c" exitCode=0
Jul 09 17:23:14.438179 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.438065    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"
Jul 09 17:23:14.438179 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.438104    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" event={"ID":"efd4345c-88fb-4e9a-a296-b209fdd7668a","Type":"ContainerDied","Data":"3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c"}
Jul 09 17:23:14.438179 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.438156    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4" event={"ID":"efd4345c-88fb-4e9a-a296-b209fdd7668a","Type":"ContainerDied","Data":"68d57bde3ae96b1a6f593d0bf3b1715559527d8887066e832dae33889e4d86f3"}
Jul 09 17:23:14.438322 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.438180    2567 scope.go:117] "RemoveContainer" containerID="3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c"
Jul 09 17:23:14.446505 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.446489    2567 scope.go:117] "RemoveContainer" containerID="3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c"
Jul 09 17:23:14.446762 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:23:14.446744    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c\": container with ID starting with 3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c not found: ID does not exist" containerID="3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c"
Jul 09 17:23:14.446826 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.446771    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c"} err="failed to get container status \"3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c\": rpc error: code = NotFound desc = could not find container \"3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c\": container with ID starting with 3663951a5607b3ee34c384c73d4031ad0f40bc7b2f4fbb7ff1c98220dd49298c not found: ID does not exist"
Jul 09 17:23:14.463208 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.463186    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59w8c\" (UniqueName: \"kubernetes.io/projected/efd4345c-88fb-4e9a-a296-b209fdd7668a-kube-api-access-59w8c\") pod \"efd4345c-88fb-4e9a-a296-b209fdd7668a\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") "
Jul 09 17:23:14.463305 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.463236    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls\") pod \"efd4345c-88fb-4e9a-a296-b209fdd7668a\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") "
Jul 09 17:23:14.463305 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.463271    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4345c-88fb-4e9a-a296-b209fdd7668a-openshift-service-ca\") pod \"efd4345c-88fb-4e9a-a296-b209fdd7668a\" (UID: \"efd4345c-88fb-4e9a-a296-b209fdd7668a\") "
Jul 09 17:23:14.463664 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.463640    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd4345c-88fb-4e9a-a296-b209fdd7668a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efd4345c-88fb-4e9a-a296-b209fdd7668a" (UID: "efd4345c-88fb-4e9a-a296-b209fdd7668a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:23:14.465270 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.465241    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd4345c-88fb-4e9a-a296-b209fdd7668a-kube-api-access-59w8c" (OuterVolumeSpecName: "kube-api-access-59w8c") pod "efd4345c-88fb-4e9a-a296-b209fdd7668a" (UID: "efd4345c-88fb-4e9a-a296-b209fdd7668a"). InnerVolumeSpecName "kube-api-access-59w8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:23:14.465270 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.465253    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efd4345c-88fb-4e9a-a296-b209fdd7668a" (UID: "efd4345c-88fb-4e9a-a296-b209fdd7668a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:23:14.564129 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.564096    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd4345c-88fb-4e9a-a296-b209fdd7668a-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:14.564129 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.564129    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd4345c-88fb-4e9a-a296-b209fdd7668a-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:14.564300 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.564141    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59w8c\" (UniqueName: \"kubernetes.io/projected/efd4345c-88fb-4e9a-a296-b209fdd7668a-kube-api-access-59w8c\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:14.758676 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.758647    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"]
Jul 09 17:23:14.762257 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:14.762236    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-33cf76-6665cd9dd9-8njq4"]
Jul 09 17:23:15.872198 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:15.872158    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd4345c-88fb-4e9a-a296-b209fdd7668a" path="/var/lib/kubelet/pods/efd4345c-88fb-4e9a-a296-b209fdd7668a/volumes"
Jul 09 17:23:17.068307 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:17.068266    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:23:23.762974 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.762944    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"]
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763250    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb054748-2013-4748-a225-55929353d4ca" containerName="main"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763260    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb054748-2013-4748-a225-55929353d4ca" containerName="main"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763272    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd4345c-88fb-4e9a-a296-b209fdd7668a" containerName="maas-api"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763278    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd4345c-88fb-4e9a-a296-b209fdd7668a" containerName="maas-api"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763290    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" containerName="main"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763296    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" containerName="main"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763306    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c6d32d-50dd-48d3-9010-9c13d26d28de" containerName="maas-api"
Jul 09 17:23:23.763332 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763311    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c6d32d-50dd-48d3-9010-9c13d26d28de" containerName="maas-api"
Jul 09 17:23:23.763577 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763366    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c1a08c-16d8-4c74-a2e0-1bfbf61365bf" containerName="main"
Jul 09 17:23:23.763577 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763377    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd4345c-88fb-4e9a-a296-b209fdd7668a" containerName="maas-api"
Jul 09 17:23:23.763577 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763386    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c6d32d-50dd-48d3-9010-9c13d26d28de" containerName="maas-api"
Jul 09 17:23:23.763577 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.763392    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb054748-2013-4748-a225-55929353d4ca" containerName="main"
Jul 09 17:23:23.767508 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.767490    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:23.770018 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.769989    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:23:23.770018 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.770011    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:23:23.771100 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.771085    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fd442b-serving-cert\""
Jul 09 17:23:23.771170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.771105    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-whstr\""
Jul 09 17:23:23.774180 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.774161    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"]
Jul 09 17:23:23.934487 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.934455    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c6f6cd-018d-4470-a513-3846cee31d73-openshift-service-ca\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:23.934636 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.934493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxs45\" (UniqueName: \"kubernetes.io/projected/12c6f6cd-018d-4470-a513-3846cee31d73-kube-api-access-vxs45\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:23.934636 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:23.934593    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c6f6cd-018d-4470-a513-3846cee31d73-maas-api-tls\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.035084 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.035001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c6f6cd-018d-4470-a513-3846cee31d73-maas-api-tls\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.035536 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.035510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c6f6cd-018d-4470-a513-3846cee31d73-openshift-service-ca\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.035672 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.035556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxs45\" (UniqueName: \"kubernetes.io/projected/12c6f6cd-018d-4470-a513-3846cee31d73-kube-api-access-vxs45\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.036438 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.036415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c6f6cd-018d-4470-a513-3846cee31d73-openshift-service-ca\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.037609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.037591    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c6f6cd-018d-4470-a513-3846cee31d73-maas-api-tls\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.064162 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.058247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxs45\" (UniqueName: \"kubernetes.io/projected/12c6f6cd-018d-4470-a513-3846cee31d73-kube-api-access-vxs45\") pod \"maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.078387 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.078363    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:24.213814 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.213702    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"]
Jul 09 17:23:24.217222 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:23:24.217190    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c6f6cd_018d_4470_a513_3846cee31d73.slice/crio-e0c14de6ce3cffc6fa0e985ef55766039db6ef635fb281c6bbc547b22903dc56 WatchSource:0}: Error finding container e0c14de6ce3cffc6fa0e985ef55766039db6ef635fb281c6bbc547b22903dc56: Status 404 returned error can't find the container with id e0c14de6ce3cffc6fa0e985ef55766039db6ef635fb281c6bbc547b22903dc56
Jul 09 17:23:24.484523 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:24.484474    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf" event={"ID":"12c6f6cd-018d-4470-a513-3846cee31d73","Type":"ContainerStarted","Data":"e0c14de6ce3cffc6fa0e985ef55766039db6ef635fb281c6bbc547b22903dc56"}
Jul 09 17:23:25.490131 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:25.490095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf" event={"ID":"12c6f6cd-018d-4470-a513-3846cee31d73","Type":"ContainerStarted","Data":"b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7"}
Jul 09 17:23:25.490575 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:25.490209    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:25.506973 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:25.506919    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf" podStartSLOduration=1.888574905 podStartE2EDuration="2.506904413s" totalImagesPullingTime="618.329508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:23:25.505900361 +0000 UTC m=+1946.203072690" watchObservedRunningTime="2026-07-09 17:23:25.506904413 +0000 UTC m=+1946.204076740"
Jul 09 17:23:31.501876 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.501842    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:31.660305 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.660268    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:23:31.664785 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.664747    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"]
Jul 09 17:23:31.668541 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.668516    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.671150 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.671129    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-890eaa-serving-cert\""
Jul 09 17:23:31.678602 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.678391    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"]
Jul 09 17:23:31.795260 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.795170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-openshift-service-ca\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.795425 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.795312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6wc\" (UniqueName: \"kubernetes.io/projected/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-kube-api-access-dt6wc\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.795425 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.795352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-maas-api-tls\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.896856 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.896825    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-openshift-service-ca\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.896989 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.896908    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6wc\" (UniqueName: \"kubernetes.io/projected/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-kube-api-access-dt6wc\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.896989 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.896945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-maas-api-tls\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.897424 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.897399    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-openshift-service-ca\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.899457 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.899440    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-maas-api-tls\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.908781 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.908759    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6wc\" (UniqueName: \"kubernetes.io/projected/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-kube-api-access-dt6wc\") pod \"maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:31.982623 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:31.982589    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:32.147149 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:32.147123    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"]
Jul 09 17:23:32.149005 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:23:32.148979    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d8cd00_e46d_4dfe_b7ad_d23753d6581a.slice/crio-956a1ca8b5a241304ed644ec5f9e668b489dded41ca00b1c29023cf732a12e37 WatchSource:0}: Error finding container 956a1ca8b5a241304ed644ec5f9e668b489dded41ca00b1c29023cf732a12e37: Status 404 returned error can't find the container with id 956a1ca8b5a241304ed644ec5f9e668b489dded41ca00b1c29023cf732a12e37
Jul 09 17:23:32.521887 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:32.521858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl" event={"ID":"13d8cd00-e46d-4dfe-b7ad-d23753d6581a","Type":"ContainerStarted","Data":"2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3"}
Jul 09 17:23:32.522193 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:32.521912    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl" event={"ID":"13d8cd00-e46d-4dfe-b7ad-d23753d6581a","Type":"ContainerStarted","Data":"956a1ca8b5a241304ed644ec5f9e668b489dded41ca00b1c29023cf732a12e37"}
Jul 09 17:23:32.522193 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:32.521995    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:32.540998 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:32.540958    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl" podStartSLOduration=1.229508834 podStartE2EDuration="1.540945054s" totalImagesPullingTime="311.43622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:23:32.539851042 +0000 UTC m=+1953.237023368" watchObservedRunningTime="2026-07-09 17:23:32.540945054 +0000 UTC m=+1953.238117415"
Jul 09 17:23:38.532112 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:38.532079    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:44.966556 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:44.966514    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"]
Jul 09 17:23:44.967054 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:44.966847    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf" podUID="12c6f6cd-018d-4470-a513-3846cee31d73" containerName="maas-api" containerID="cri-o://b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7" gracePeriod=30
Jul 09 17:23:45.210524 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.210495    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:45.300389 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.300302    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxs45\" (UniqueName: \"kubernetes.io/projected/12c6f6cd-018d-4470-a513-3846cee31d73-kube-api-access-vxs45\") pod \"12c6f6cd-018d-4470-a513-3846cee31d73\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") "
Jul 09 17:23:45.300389 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.300368    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c6f6cd-018d-4470-a513-3846cee31d73-openshift-service-ca\") pod \"12c6f6cd-018d-4470-a513-3846cee31d73\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") "
Jul 09 17:23:45.300607 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.300446    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c6f6cd-018d-4470-a513-3846cee31d73-maas-api-tls\") pod \"12c6f6cd-018d-4470-a513-3846cee31d73\" (UID: \"12c6f6cd-018d-4470-a513-3846cee31d73\") "
Jul 09 17:23:45.300683 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.300658    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c6f6cd-018d-4470-a513-3846cee31d73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12c6f6cd-018d-4470-a513-3846cee31d73" (UID: "12c6f6cd-018d-4470-a513-3846cee31d73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:23:45.302509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.302479    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c6f6cd-018d-4470-a513-3846cee31d73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12c6f6cd-018d-4470-a513-3846cee31d73" (UID: "12c6f6cd-018d-4470-a513-3846cee31d73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:23:45.302622 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.302581    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c6f6cd-018d-4470-a513-3846cee31d73-kube-api-access-vxs45" (OuterVolumeSpecName: "kube-api-access-vxs45") pod "12c6f6cd-018d-4470-a513-3846cee31d73" (UID: "12c6f6cd-018d-4470-a513-3846cee31d73"). InnerVolumeSpecName "kube-api-access-vxs45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:23:45.401609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.401571    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c6f6cd-018d-4470-a513-3846cee31d73-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:45.401609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.401600    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c6f6cd-018d-4470-a513-3846cee31d73-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:45.401609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.401614    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxs45\" (UniqueName: \"kubernetes.io/projected/12c6f6cd-018d-4470-a513-3846cee31d73-kube-api-access-vxs45\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:45.580287 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.580210    2567 generic.go:356] "Generic (PLEG): container finished" podID="12c6f6cd-018d-4470-a513-3846cee31d73" containerID="b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7" exitCode=0
Jul 09 17:23:45.580287 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.580256    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"
Jul 09 17:23:45.580470 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.580298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf" event={"ID":"12c6f6cd-018d-4470-a513-3846cee31d73","Type":"ContainerDied","Data":"b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7"}
Jul 09 17:23:45.580470 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.580351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf" event={"ID":"12c6f6cd-018d-4470-a513-3846cee31d73","Type":"ContainerDied","Data":"e0c14de6ce3cffc6fa0e985ef55766039db6ef635fb281c6bbc547b22903dc56"}
Jul 09 17:23:45.580470 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.580373    2567 scope.go:117] "RemoveContainer" containerID="b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7"
Jul 09 17:23:45.589235 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.589219    2567 scope.go:117] "RemoveContainer" containerID="b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7"
Jul 09 17:23:45.589488 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:23:45.589473    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7\": container with ID starting with b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7 not found: ID does not exist" containerID="b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7"
Jul 09 17:23:45.589535 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.589497    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7"} err="failed to get container status \"b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7\": rpc error: code = NotFound desc = could not find container \"b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7\": container with ID starting with b3aff0f0284c384f704baa6c7f8fd2987d94cf0e8cb7627e1bd42f41e2c1aca7 not found: ID does not exist"
Jul 09 17:23:45.601237 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.601209    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"]
Jul 09 17:23:45.607660 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.607630    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fd442b-5dcd54d95f-ffkrf"]
Jul 09 17:23:45.872027 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:45.871997    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c6f6cd-018d-4470-a513-3846cee31d73" path="/var/lib/kubelet/pods/12c6f6cd-018d-4470-a513-3846cee31d73/volumes"
Jul 09 17:23:46.163223 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:46.163139    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:23:52.102648 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.102455    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"]
Jul 09 17:23:52.103491 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.103434    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl" podUID="13d8cd00-e46d-4dfe-b7ad-d23753d6581a" containerName="maas-api" containerID="cri-o://2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3" gracePeriod=30
Jul 09 17:23:52.361655 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.361598    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:52.458823 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.458767    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-openshift-service-ca\") pod \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") "
Jul 09 17:23:52.458977 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.458884    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt6wc\" (UniqueName: \"kubernetes.io/projected/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-kube-api-access-dt6wc\") pod \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") "
Jul 09 17:23:52.458977 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.458920    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-maas-api-tls\") pod \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\" (UID: \"13d8cd00-e46d-4dfe-b7ad-d23753d6581a\") "
Jul 09 17:23:52.459088 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.459068    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13d8cd00-e46d-4dfe-b7ad-d23753d6581a" (UID: "13d8cd00-e46d-4dfe-b7ad-d23753d6581a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:23:52.459177 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.459161    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:52.461172 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.461140    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13d8cd00-e46d-4dfe-b7ad-d23753d6581a" (UID: "13d8cd00-e46d-4dfe-b7ad-d23753d6581a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:23:52.461172 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.461150    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-kube-api-access-dt6wc" (OuterVolumeSpecName: "kube-api-access-dt6wc") pod "13d8cd00-e46d-4dfe-b7ad-d23753d6581a" (UID: "13d8cd00-e46d-4dfe-b7ad-d23753d6581a"). InnerVolumeSpecName "kube-api-access-dt6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:23:52.560100 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.560074    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6wc\" (UniqueName: \"kubernetes.io/projected/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-kube-api-access-dt6wc\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:52.560100 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.560097    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d8cd00-e46d-4dfe-b7ad-d23753d6581a-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:23:52.614704 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.614645    2567 generic.go:356] "Generic (PLEG): container finished" podID="13d8cd00-e46d-4dfe-b7ad-d23753d6581a" containerID="2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3" exitCode=0
Jul 09 17:23:52.614704 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.614696    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"
Jul 09 17:23:52.614849 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.614727    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl" event={"ID":"13d8cd00-e46d-4dfe-b7ad-d23753d6581a","Type":"ContainerDied","Data":"2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3"}
Jul 09 17:23:52.614849 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.614777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl" event={"ID":"13d8cd00-e46d-4dfe-b7ad-d23753d6581a","Type":"ContainerDied","Data":"956a1ca8b5a241304ed644ec5f9e668b489dded41ca00b1c29023cf732a12e37"}
Jul 09 17:23:52.614849 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.614811    2567 scope.go:117] "RemoveContainer" containerID="2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3"
Jul 09 17:23:52.623566 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.623551    2567 scope.go:117] "RemoveContainer" containerID="2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3"
Jul 09 17:23:52.623851 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:23:52.623822    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3\": container with ID starting with 2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3 not found: ID does not exist" containerID="2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3"
Jul 09 17:23:52.623941 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.623859    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3"} err="failed to get container status \"2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3\": rpc error: code = NotFound desc = could not find container \"2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3\": container with ID starting with 2ba40ccef63192a3c460e53c7fe9fb26fa6d93a3d0e07d7870c3a126386ce0f3 not found: ID does not exist"
Jul 09 17:23:52.644617 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.644595    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"]
Jul 09 17:23:52.648665 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:52.648644    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-890eaa-f5b99d4cd-6spwl"]
Jul 09 17:23:53.872025 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:53.871995    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d8cd00-e46d-4dfe-b7ad-d23753d6581a" path="/var/lib/kubelet/pods/13d8cd00-e46d-4dfe-b7ad-d23753d6581a/volumes"
Jul 09 17:23:56.754052 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:23:56.754007    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:24:01.884143 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884115    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"]
Jul 09 17:24:01.884599 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884430    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d8cd00-e46d-4dfe-b7ad-d23753d6581a" containerName="maas-api"
Jul 09 17:24:01.884599 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884439    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d8cd00-e46d-4dfe-b7ad-d23753d6581a" containerName="maas-api"
Jul 09 17:24:01.884599 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884460    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c6f6cd-018d-4470-a513-3846cee31d73" containerName="maas-api"
Jul 09 17:24:01.884599 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884466    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c6f6cd-018d-4470-a513-3846cee31d73" containerName="maas-api"
Jul 09 17:24:01.884599 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884513    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c6f6cd-018d-4470-a513-3846cee31d73" containerName="maas-api"
Jul 09 17:24:01.884599 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.884522    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d8cd00-e46d-4dfe-b7ad-d23753d6581a" containerName="maas-api"
Jul 09 17:24:01.887509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.887379    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:01.890439 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.890414    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:24:01.890547 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.890437    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:24:01.890547 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.890415    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-whstr\""
Jul 09 17:24:01.890725 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.890712    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-0abbe5-serving-cert\""
Jul 09 17:24:01.902991 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.902966    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"]
Jul 09 17:24:01.936172 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.936149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be88f08-ccf7-4ef2-97a1-239925442c3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:01.936267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.936186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be88f08-ccf7-4ef2-97a1-239925442c3f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:01.936267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:01.936216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjgs\" (UniqueName: \"kubernetes.io/projected/4be88f08-ccf7-4ef2-97a1-239925442c3f-kube-api-access-qqjgs\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.037174 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.037143    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be88f08-ccf7-4ef2-97a1-239925442c3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.037267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.037181    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be88f08-ccf7-4ef2-97a1-239925442c3f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.037267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.037228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjgs\" (UniqueName: \"kubernetes.io/projected/4be88f08-ccf7-4ef2-97a1-239925442c3f-kube-api-access-qqjgs\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.037862 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.037843    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be88f08-ccf7-4ef2-97a1-239925442c3f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.039463 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.039443    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be88f08-ccf7-4ef2-97a1-239925442c3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.048415 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.048391    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjgs\" (UniqueName: \"kubernetes.io/projected/4be88f08-ccf7-4ef2-97a1-239925442c3f-kube-api-access-qqjgs\") pod \"maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.198232 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.198181    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:02.330565 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.330536    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"]
Jul 09 17:24:02.332944 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:24:02.332918    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be88f08_ccf7_4ef2_97a1_239925442c3f.slice/crio-e5a2b2070499b9e47056363b7f8d02b43a12b7029b0be7ac35d1b9666cae1902 WatchSource:0}: Error finding container e5a2b2070499b9e47056363b7f8d02b43a12b7029b0be7ac35d1b9666cae1902: Status 404 returned error can't find the container with id e5a2b2070499b9e47056363b7f8d02b43a12b7029b0be7ac35d1b9666cae1902
Jul 09 17:24:02.659482 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:02.659449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5" event={"ID":"4be88f08-ccf7-4ef2-97a1-239925442c3f","Type":"ContainerStarted","Data":"e5a2b2070499b9e47056363b7f8d02b43a12b7029b0be7ac35d1b9666cae1902"}
Jul 09 17:24:03.665104 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:03.665069    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5" event={"ID":"4be88f08-ccf7-4ef2-97a1-239925442c3f","Type":"ContainerStarted","Data":"15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea"}
Jul 09 17:24:03.665573 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:03.665162    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:03.682631 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:03.682587    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5" podStartSLOduration=2.308620204 podStartE2EDuration="2.682574632s" totalImagesPullingTime="373.954428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:24:03.680581641 +0000 UTC m=+1984.377753967" watchObservedRunningTime="2026-07-09 17:24:03.682574632 +0000 UTC m=+1984.379746957"
Jul 09 17:24:09.257882 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:09.257847    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:24:09.675185 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:09.675151    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:24:10.054288 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.054180    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"]
Jul 09 17:24:10.057598 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.057574    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.060410 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.060387    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-265f70-serving-cert\""
Jul 09 17:24:10.069905 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.069876    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"]
Jul 09 17:24:10.103733 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.103705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b186b48-a909-4fe7-985c-03337caff987-openshift-service-ca\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.103934 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.103776    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b186b48-a909-4fe7-985c-03337caff987-maas-api-tls\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.103934 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.103865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmj96\" (UniqueName: \"kubernetes.io/projected/0b186b48-a909-4fe7-985c-03337caff987-kube-api-access-cmj96\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.205241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.205204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmj96\" (UniqueName: \"kubernetes.io/projected/0b186b48-a909-4fe7-985c-03337caff987-kube-api-access-cmj96\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.205434 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.205258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b186b48-a909-4fe7-985c-03337caff987-openshift-service-ca\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.205434 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.205318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b186b48-a909-4fe7-985c-03337caff987-maas-api-tls\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.205966 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.205942    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b186b48-a909-4fe7-985c-03337caff987-openshift-service-ca\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.208266 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.208232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b186b48-a909-4fe7-985c-03337caff987-maas-api-tls\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.215073 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.215049    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmj96\" (UniqueName: \"kubernetes.io/projected/0b186b48-a909-4fe7-985c-03337caff987-kube-api-access-cmj96\") pod \"maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.371615 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.371584    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:10.506134 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.506099    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"]
Jul 09 17:24:10.509689 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:24:10.509645    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b186b48_a909_4fe7_985c_03337caff987.slice/crio-6ae2e0f6fa5a934e2b5c047e292d406017d14ecbd6bd96a955822b7aea066c1b WatchSource:0}: Error finding container 6ae2e0f6fa5a934e2b5c047e292d406017d14ecbd6bd96a955822b7aea066c1b: Status 404 returned error can't find the container with id 6ae2e0f6fa5a934e2b5c047e292d406017d14ecbd6bd96a955822b7aea066c1b
Jul 09 17:24:10.695908 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:10.695829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k" event={"ID":"0b186b48-a909-4fe7-985c-03337caff987","Type":"ContainerStarted","Data":"6ae2e0f6fa5a934e2b5c047e292d406017d14ecbd6bd96a955822b7aea066c1b"}
Jul 09 17:24:11.702979 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:11.702943    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k" event={"ID":"0b186b48-a909-4fe7-985c-03337caff987","Type":"ContainerStarted","Data":"ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7"}
Jul 09 17:24:11.703516 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:11.703027    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:11.721117 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:11.721068    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k" podStartSLOduration=1.392103039 podStartE2EDuration="1.721053652s" totalImagesPullingTime="328.950613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:24:11.719164129 +0000 UTC m=+1992.416336456" watchObservedRunningTime="2026-07-09 17:24:11.721053652 +0000 UTC m=+1992.418225996"
Jul 09 17:24:17.712042 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:17.712013    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:24:20.360726 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:20.360692    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:24:30.541209 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.541175    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"]
Jul 09 17:24:30.544905 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.544870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.547533 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.547511    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"openshift-service-ca.crt\""
Jul 09 17:24:30.547655 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.547512    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"default-dockercfg-ckk8f\""
Jul 09 17:24:30.548774 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.548755    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"auth-test-model-0abbe5-kserve-self-signed-certs\""
Jul 09 17:24:30.548920 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.548853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"kube-root-ca.crt\""
Jul 09 17:24:30.555778 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.555756    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"]
Jul 09 17:24:30.666268 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.666242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-model-cache\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.666380 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.666279    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-dshm\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.666380 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.666313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-tmp-dir\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.666380 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.666349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f427db3-6397-4a77-ac0d-2b0a753f71de-tls-certs\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.666497 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.666421    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8ln\" (UniqueName: \"kubernetes.io/projected/8f427db3-6397-4a77-ac0d-2b0a753f71de-kube-api-access-tx8ln\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.666497 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.666454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-home\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.767753 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.767720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-model-cache\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.767895 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.767774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-dshm\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.767895 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.767846    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-tmp-dir\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.767895 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.767869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f427db3-6397-4a77-ac0d-2b0a753f71de-tls-certs\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.768075 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.767898    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8ln\" (UniqueName: \"kubernetes.io/projected/8f427db3-6397-4a77-ac0d-2b0a753f71de-kube-api-access-tx8ln\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.768075 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.768028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-home\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.768194 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.768172    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-model-cache\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.768265 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.768233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-tmp-dir\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.768356 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.768337    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-home\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.770246 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.770226    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-dshm\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.770420 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.770405    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f427db3-6397-4a77-ac0d-2b0a753f71de-tls-certs\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.776178 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.776159    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8ln\" (UniqueName: \"kubernetes.io/projected/8f427db3-6397-4a77-ac0d-2b0a753f71de-kube-api-access-tx8ln\") pod \"auth-test-model-0abbe5-kserve-55997cf6c-gqsnq\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") " pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.854929 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.854900    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:30.991146 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:30.991109    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"]
Jul 09 17:24:30.994033 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:24:30.994004    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f427db3_6397_4a77_ac0d_2b0a753f71de.slice/crio-37d8784e7b2c2cbb32f76e3f8d26b20cf8ab1ecf554d2e7ad509b780996c462b WatchSource:0}: Error finding container 37d8784e7b2c2cbb32f76e3f8d26b20cf8ab1ecf554d2e7ad509b780996c462b: Status 404 returned error can't find the container with id 37d8784e7b2c2cbb32f76e3f8d26b20cf8ab1ecf554d2e7ad509b780996c462b
Jul 09 17:24:31.791589 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:31.791550    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq" event={"ID":"8f427db3-6397-4a77-ac0d-2b0a753f71de","Type":"ContainerStarted","Data":"167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612"}
Jul 09 17:24:31.792002 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:31.791598    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq" event={"ID":"8f427db3-6397-4a77-ac0d-2b0a753f71de","Type":"ContainerStarted","Data":"37d8784e7b2c2cbb32f76e3f8d26b20cf8ab1ecf554d2e7ad509b780996c462b"}
Jul 09 17:24:31.810515 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:31.810469    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq" podStartSLOduration=1.810454166 podStartE2EDuration="1.810454166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:24:31.809019787 +0000 UTC m=+2012.506192127" watchObservedRunningTime="2026-07-09 17:24:31.810454166 +0000 UTC m=+2012.507626492"
Jul 09 17:24:33.762212 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:33.762176    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:24:40.855112 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:40.855080    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:40.855545 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:40.855318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:40.867654 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:40.867635    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:41.849961 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:41.849934    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:24:59.165039 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.165001    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"]
Jul 09 17:24:59.169574 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.169556    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.172667 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.172645    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"openshift-service-ca.crt\""
Jul 09 17:24:59.172791 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.172707    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"default-dockercfg-rctq9\""
Jul 09 17:24:59.173863 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.173846    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"kube-root-ca.crt\""
Jul 09 17:24:59.176575 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.174327    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"auth-test-model-265f70-kserve-self-signed-certs\""
Jul 09 17:24:59.185069 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.185047    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"]
Jul 09 17:24:59.198785 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.198762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-model-cache\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.198914 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.198830    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-dshm\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.198914 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.198862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwqb\" (UniqueName: \"kubernetes.io/projected/f828b869-94d4-4e9f-95a2-91f41c57ba70-kube-api-access-txwqb\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.198914 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.198893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-home\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.198914 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.198911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f828b869-94d4-4e9f-95a2-91f41c57ba70-tls-certs\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.199050 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.198963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-tmp-dir\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.299989 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.299950    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-home\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.299998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f828b869-94d4-4e9f-95a2-91f41c57ba70-tls-certs\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-tmp-dir\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-model-cache\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300139    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-dshm\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300375 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwqb\" (UniqueName: \"kubernetes.io/projected/f828b869-94d4-4e9f-95a2-91f41c57ba70-kube-api-access-txwqb\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300504 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-home\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300583 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300525    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-tmp-dir\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.300583 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.300537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-model-cache\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.302480 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.302463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-dshm\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.302933 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.302915    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f828b869-94d4-4e9f-95a2-91f41c57ba70-tls-certs\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.310660 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.310629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwqb\" (UniqueName: \"kubernetes.io/projected/f828b869-94d4-4e9f-95a2-91f41c57ba70-kube-api-access-txwqb\") pod \"auth-test-model-265f70-kserve-6c45776c9f-nh7r5\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") " pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.485500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.485414    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:24:59.640190 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.640157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"]
Jul 09 17:24:59.643463 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:24:59.643433    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf828b869_94d4_4e9f_95a2_91f41c57ba70.slice/crio-b8f5434547e015415bfa5bb0ca7294b837ebc835a92a5476fd347512ebc6066c WatchSource:0}: Error finding container b8f5434547e015415bfa5bb0ca7294b837ebc835a92a5476fd347512ebc6066c: Status 404 returned error can't find the container with id b8f5434547e015415bfa5bb0ca7294b837ebc835a92a5476fd347512ebc6066c
Jul 09 17:24:59.921090 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.921058    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5" event={"ID":"f828b869-94d4-4e9f-95a2-91f41c57ba70","Type":"ContainerStarted","Data":"a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679"}
Jul 09 17:24:59.921090 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.921098    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5" event={"ID":"f828b869-94d4-4e9f-95a2-91f41c57ba70","Type":"ContainerStarted","Data":"b8f5434547e015415bfa5bb0ca7294b837ebc835a92a5476fd347512ebc6066c"}
Jul 09 17:24:59.941840 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:24:59.941779    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5" podStartSLOduration=0.941764899 podStartE2EDuration="941.764899ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:24:59.940679537 +0000 UTC m=+2040.637851865" watchObservedRunningTime="2026-07-09 17:24:59.941764899 +0000 UTC m=+2040.638937226"
Jul 09 17:25:03.257771 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:03.257733    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:25:09.486396 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:09.486357    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:25:09.486790 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:09.486427    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:25:09.499244 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:09.499224    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:25:09.978391 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:09.978365    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:25:17.361809 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:17.361750    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:25:31.969340 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:31.969264    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:25:46.854300 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:25:46.854262    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:26:05.656773 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:26:05.656726    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:26:24.258334 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:26:24.258300    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:26:38.459325 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:26:38.459256    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:26:55.459275 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:26:55.459197    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:26:59.649675 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:26:59.649642    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"]
Jul 09 17:27:00.439303 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:00.439273    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"]
Jul 09 17:27:13.657911 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:13.657865    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:27:14.669732 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:14.669688    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq" podUID="8f427db3-6397-4a77-ac0d-2b0a753f71de" containerName="main" containerID="cri-o://167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612" gracePeriod=45
Jul 09 17:27:14.904982 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:14.904958    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:27:15.067081 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067014    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-tmp-dir\") pod \"8f427db3-6397-4a77-ac0d-2b0a753f71de\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") "
Jul 09 17:27:15.067081 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067047    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-home\") pod \"8f427db3-6397-4a77-ac0d-2b0a753f71de\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") "
Jul 09 17:27:15.067261 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067084    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-dshm\") pod \"8f427db3-6397-4a77-ac0d-2b0a753f71de\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") "
Jul 09 17:27:15.067261 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067115    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-model-cache\") pod \"8f427db3-6397-4a77-ac0d-2b0a753f71de\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") "
Jul 09 17:27:15.067261 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067173    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f427db3-6397-4a77-ac0d-2b0a753f71de-tls-certs\") pod \"8f427db3-6397-4a77-ac0d-2b0a753f71de\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") "
Jul 09 17:27:15.067261 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067236    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx8ln\" (UniqueName: \"kubernetes.io/projected/8f427db3-6397-4a77-ac0d-2b0a753f71de-kube-api-access-tx8ln\") pod \"8f427db3-6397-4a77-ac0d-2b0a753f71de\" (UID: \"8f427db3-6397-4a77-ac0d-2b0a753f71de\") "
Jul 09 17:27:15.067459 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067299    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-home" (OuterVolumeSpecName: "home") pod "8f427db3-6397-4a77-ac0d-2b0a753f71de" (UID: "8f427db3-6397-4a77-ac0d-2b0a753f71de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.067459 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067305    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f427db3-6397-4a77-ac0d-2b0a753f71de" (UID: "8f427db3-6397-4a77-ac0d-2b0a753f71de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.067459 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067372    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f427db3-6397-4a77-ac0d-2b0a753f71de" (UID: "8f427db3-6397-4a77-ac0d-2b0a753f71de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.067609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067475    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.067609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067507    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.067609 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.067515    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.069467 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.069440    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-dshm" (OuterVolumeSpecName: "dshm") pod "8f427db3-6397-4a77-ac0d-2b0a753f71de" (UID: "8f427db3-6397-4a77-ac0d-2b0a753f71de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.069562 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.069469    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f427db3-6397-4a77-ac0d-2b0a753f71de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f427db3-6397-4a77-ac0d-2b0a753f71de" (UID: "8f427db3-6397-4a77-ac0d-2b0a753f71de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:27:15.069562 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.069471    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f427db3-6397-4a77-ac0d-2b0a753f71de-kube-api-access-tx8ln" (OuterVolumeSpecName: "kube-api-access-tx8ln") pod "8f427db3-6397-4a77-ac0d-2b0a753f71de" (UID: "8f427db3-6397-4a77-ac0d-2b0a753f71de"). InnerVolumeSpecName "kube-api-access-tx8ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:27:15.168314 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.168291    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx8ln\" (UniqueName: \"kubernetes.io/projected/8f427db3-6397-4a77-ac0d-2b0a753f71de-kube-api-access-tx8ln\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.168314 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.168312    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f427db3-6397-4a77-ac0d-2b0a753f71de-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.168440 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.168323    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f427db3-6397-4a77-ac0d-2b0a753f71de-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.460222 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.460181    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5" podUID="f828b869-94d4-4e9f-95a2-91f41c57ba70" containerName="main" containerID="cri-o://a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679" gracePeriod=45
Jul 09 17:27:15.540157 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.540130    2567 generic.go:356] "Generic (PLEG): container finished" podID="8f427db3-6397-4a77-ac0d-2b0a753f71de" containerID="167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612" exitCode=0
Jul 09 17:27:15.540253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.540172    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"
Jul 09 17:27:15.540253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.540194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq" event={"ID":"8f427db3-6397-4a77-ac0d-2b0a753f71de","Type":"ContainerDied","Data":"167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612"}
Jul 09 17:27:15.540253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.540231    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq" event={"ID":"8f427db3-6397-4a77-ac0d-2b0a753f71de","Type":"ContainerDied","Data":"37d8784e7b2c2cbb32f76e3f8d26b20cf8ab1ecf554d2e7ad509b780996c462b"}
Jul 09 17:27:15.540253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.540250    2567 scope.go:117] "RemoveContainer" containerID="167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612"
Jul 09 17:27:15.560927 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.560904    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"]
Jul 09 17:27:15.564787 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.564766    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/auth-test-model-0abbe5-kserve-55997cf6c-gqsnq"]
Jul 09 17:27:15.570857 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.570830    2567 scope.go:117] "RemoveContainer" containerID="167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612"
Jul 09 17:27:15.571147 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:27:15.571118    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612\": container with ID starting with 167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612 not found: ID does not exist" containerID="167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612"
Jul 09 17:27:15.571233 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.571164    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612"} err="failed to get container status \"167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612\": rpc error: code = NotFound desc = could not find container \"167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612\": container with ID starting with 167555370a66d8b255b813f81955c13d384a1ea97e6ec0016c0a4dcb36bb5612 not found: ID does not exist"
Jul 09 17:27:15.713470 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.713410    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:27:15.872125 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.872098    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f427db3-6397-4a77-ac0d-2b0a753f71de" path="/var/lib/kubelet/pods/8f427db3-6397-4a77-ac0d-2b0a753f71de/volumes"
Jul 09 17:27:15.873676 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.873659    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txwqb\" (UniqueName: \"kubernetes.io/projected/f828b869-94d4-4e9f-95a2-91f41c57ba70-kube-api-access-txwqb\") pod \"f828b869-94d4-4e9f-95a2-91f41c57ba70\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") "
Jul 09 17:27:15.873747 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.873718    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f828b869-94d4-4e9f-95a2-91f41c57ba70-tls-certs\") pod \"f828b869-94d4-4e9f-95a2-91f41c57ba70\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") "
Jul 09 17:27:15.873785 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.873754    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-home\") pod \"f828b869-94d4-4e9f-95a2-91f41c57ba70\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") "
Jul 09 17:27:15.873785 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.873779    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-tmp-dir\") pod \"f828b869-94d4-4e9f-95a2-91f41c57ba70\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") "
Jul 09 17:27:15.873898 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.873818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-dshm\") pod \"f828b869-94d4-4e9f-95a2-91f41c57ba70\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") "
Jul 09 17:27:15.873898 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.873876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-model-cache\") pod \"f828b869-94d4-4e9f-95a2-91f41c57ba70\" (UID: \"f828b869-94d4-4e9f-95a2-91f41c57ba70\") "
Jul 09 17:27:15.874081 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.874054    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-home" (OuterVolumeSpecName: "home") pod "f828b869-94d4-4e9f-95a2-91f41c57ba70" (UID: "f828b869-94d4-4e9f-95a2-91f41c57ba70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.874244 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.874222    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-model-cache" (OuterVolumeSpecName: "model-cache") pod "f828b869-94d4-4e9f-95a2-91f41c57ba70" (UID: "f828b869-94d4-4e9f-95a2-91f41c57ba70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.874301 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.874066    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f828b869-94d4-4e9f-95a2-91f41c57ba70" (UID: "f828b869-94d4-4e9f-95a2-91f41c57ba70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.875992 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.875968    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f828b869-94d4-4e9f-95a2-91f41c57ba70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f828b869-94d4-4e9f-95a2-91f41c57ba70" (UID: "f828b869-94d4-4e9f-95a2-91f41c57ba70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:27:15.876358 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.876339    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f828b869-94d4-4e9f-95a2-91f41c57ba70-kube-api-access-txwqb" (OuterVolumeSpecName: "kube-api-access-txwqb") pod "f828b869-94d4-4e9f-95a2-91f41c57ba70" (UID: "f828b869-94d4-4e9f-95a2-91f41c57ba70"). InnerVolumeSpecName "kube-api-access-txwqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:27:15.876358 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.876344    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-dshm" (OuterVolumeSpecName: "dshm") pod "f828b869-94d4-4e9f-95a2-91f41c57ba70" (UID: "f828b869-94d4-4e9f-95a2-91f41c57ba70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:15.974509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.974454    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwqb\" (UniqueName: \"kubernetes.io/projected/f828b869-94d4-4e9f-95a2-91f41c57ba70-kube-api-access-txwqb\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.974509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.974472    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f828b869-94d4-4e9f-95a2-91f41c57ba70-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.974509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.974481    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.974509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.974489    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.974509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.974496    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:15.974509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:15.974503    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f828b869-94d4-4e9f-95a2-91f41c57ba70-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:27:16.546245 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.546207    2567 generic.go:356] "Generic (PLEG): container finished" podID="f828b869-94d4-4e9f-95a2-91f41c57ba70" containerID="a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679" exitCode=0
Jul 09 17:27:16.546438 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.546249    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"
Jul 09 17:27:16.546438 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.546288    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5" event={"ID":"f828b869-94d4-4e9f-95a2-91f41c57ba70","Type":"ContainerDied","Data":"a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679"}
Jul 09 17:27:16.546438 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.546330    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5" event={"ID":"f828b869-94d4-4e9f-95a2-91f41c57ba70","Type":"ContainerDied","Data":"b8f5434547e015415bfa5bb0ca7294b837ebc835a92a5476fd347512ebc6066c"}
Jul 09 17:27:16.546438 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.546349    2567 scope.go:117] "RemoveContainer" containerID="a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679"
Jul 09 17:27:16.555273 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.555257    2567 scope.go:117] "RemoveContainer" containerID="a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679"
Jul 09 17:27:16.555530 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:27:16.555514    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679\": container with ID starting with a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679 not found: ID does not exist" containerID="a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679"
Jul 09 17:27:16.555575 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.555539    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679"} err="failed to get container status \"a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679\": rpc error: code = NotFound desc = could not find container \"a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679\": container with ID starting with a4c1ce120bbf607a16a6015652075d9dc5c2878b75279a9138e91762c607c679 not found: ID does not exist"
Jul 09 17:27:16.567611 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.567589    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"]
Jul 09 17:27:16.575106 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:16.575088    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/auth-test-model-265f70-kserve-6c45776c9f-nh7r5"]
Jul 09 17:27:17.871925 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:17.871897    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f828b869-94d4-4e9f-95a2-91f41c57ba70" path="/var/lib/kubelet/pods/f828b869-94d4-4e9f-95a2-91f41c57ba70/volumes"
Jul 09 17:27:18.457291 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457257    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"]
Jul 09 17:27:18.457619 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457606    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f427db3-6397-4a77-ac0d-2b0a753f71de" containerName="main"
Jul 09 17:27:18.457658 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457621    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f427db3-6397-4a77-ac0d-2b0a753f71de" containerName="main"
Jul 09 17:27:18.457658 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457635    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f828b869-94d4-4e9f-95a2-91f41c57ba70" containerName="main"
Jul 09 17:27:18.457658 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457641    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f828b869-94d4-4e9f-95a2-91f41c57ba70" containerName="main"
Jul 09 17:27:18.457751 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457698    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f427db3-6397-4a77-ac0d-2b0a753f71de" containerName="main"
Jul 09 17:27:18.457751 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.457708    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f828b869-94d4-4e9f-95a2-91f41c57ba70" containerName="main"
Jul 09 17:27:18.461965 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.461945    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.464516 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.464496    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"kube-root-ca.crt\""
Jul 09 17:27:18.465717 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.465696    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"openshift-service-ca.crt\""
Jul 09 17:27:18.465889 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.465736    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"default-dockercfg-ckk8f\""
Jul 09 17:27:18.465889 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.465763    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"sub-test-model-0abbe5-kserve-self-signed-certs\""
Jul 09 17:27:18.471066 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.471048    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"]
Jul 09 17:27:18.592198 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.592166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-tmp-dir\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.592343 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.592221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-home\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.592343 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.592315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538f2879-3b82-4079-991b-8b503b05588f-tls-certs\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.592449 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.592348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-model-cache\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.592492 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.592462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf497\" (UniqueName: \"kubernetes.io/projected/538f2879-3b82-4079-991b-8b503b05588f-kube-api-access-xf497\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.592538 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.592510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-dshm\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.693711 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.693677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf497\" (UniqueName: \"kubernetes.io/projected/538f2879-3b82-4079-991b-8b503b05588f-kube-api-access-xf497\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.693892 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.693731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-dshm\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.693892 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.693780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-tmp-dir\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.693892 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.693833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-home\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.693892 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.693874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538f2879-3b82-4079-991b-8b503b05588f-tls-certs\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.694080 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.693897    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-model-cache\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.694244 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.694212    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-tmp-dir\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.694309 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.694281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-model-cache\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.694309 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.694300    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-home\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.696463 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.696430    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-dshm\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.696463 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.696457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538f2879-3b82-4079-991b-8b503b05588f-tls-certs\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.701614 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.701591    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf497\" (UniqueName: \"kubernetes.io/projected/538f2879-3b82-4079-991b-8b503b05588f-kube-api-access-xf497\") pod \"sub-test-model-0abbe5-kserve-7bddf86f85-72crx\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") " pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.772712 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.772653    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:18.904543 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:18.904519    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"]
Jul 09 17:27:18.907055 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:27:18.907029    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod538f2879_3b82_4079_991b_8b503b05588f.slice/crio-4b0fa7c469f11d018ddb23c4c3aa012382965da0dd62faf03bbe380db2554a1e WatchSource:0}: Error finding container 4b0fa7c469f11d018ddb23c4c3aa012382965da0dd62faf03bbe380db2554a1e: Status 404 returned error can't find the container with id 4b0fa7c469f11d018ddb23c4c3aa012382965da0dd62faf03bbe380db2554a1e
Jul 09 17:27:19.563528 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:19.563497    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx" event={"ID":"538f2879-3b82-4079-991b-8b503b05588f","Type":"ContainerStarted","Data":"d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b"}
Jul 09 17:27:19.563720 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:19.563536    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx" event={"ID":"538f2879-3b82-4079-991b-8b503b05588f","Type":"ContainerStarted","Data":"4b0fa7c469f11d018ddb23c4c3aa012382965da0dd62faf03bbe380db2554a1e"}
Jul 09 17:27:19.581441 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:19.581400    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx" podStartSLOduration=1.581385864 podStartE2EDuration="1.581385864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:27:19.580346389 +0000 UTC m=+2180.277518716" watchObservedRunningTime="2026-07-09 17:27:19.581385864 +0000 UTC m=+2180.278558188"
Jul 09 17:27:22.562434 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:22.562390    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:27:28.773375 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:28.773337    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:28.773375 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:28.773383    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:28.785514 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:28.785489    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:29.620331 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:29.620297    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:27:42.837902 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.837843    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"]
Jul 09 17:27:42.841303 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.841280    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.845925 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.845898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"openshift-service-ca.crt\""
Jul 09 17:27:42.846077 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.845934    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"default-dockercfg-rctq9\""
Jul 09 17:27:42.846077 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.845948    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"sub-test-model-265f70-kserve-self-signed-certs\""
Jul 09 17:27:42.846077 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.846062    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"kube-root-ca.crt\""
Jul 09 17:27:42.850050 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.850027    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"]
Jul 09 17:27:42.887698 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.887673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tmp-dir\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.887921 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.887730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-home\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.887921 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.887878    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-dshm\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.887921 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.887902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp942\" (UniqueName: \"kubernetes.io/projected/ed0df847-ea7c-4ed4-9f5e-67543b25a753-kube-api-access-pp942\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.888084 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.887939    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-model-cache\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.888084 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.887964    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tls-certs\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.988729 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.988704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-dshm\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.988894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.988730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp942\" (UniqueName: \"kubernetes.io/projected/ed0df847-ea7c-4ed4-9f5e-67543b25a753-kube-api-access-pp942\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.988894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.988754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-model-cache\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.988894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.988772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tls-certs\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.989070 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.988927    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tmp-dir\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.989070 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.988984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-home\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.989200 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.989184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-model-cache\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.989275 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.989254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tmp-dir\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.989323 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.989285    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-home\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.990892 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.990873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-dshm\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.991200 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.991183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tls-certs\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:42.999235 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:42.999216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp942\" (UniqueName: \"kubernetes.io/projected/ed0df847-ea7c-4ed4-9f5e-67543b25a753-kube-api-access-pp942\") pod \"sub-test-model-265f70-kserve-84ccdfdccf-24phz\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") " pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:43.151343 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:43.151320    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:43.281582 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:43.281559    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"]
Jul 09 17:27:43.283042 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:27:43.283016    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded0df847_ea7c_4ed4_9f5e_67543b25a753.slice/crio-4de3b47b3ee24c4b18ed6ff54307868d4caf83a0107a7a87877ab78ba35292ef WatchSource:0}: Error finding container 4de3b47b3ee24c4b18ed6ff54307868d4caf83a0107a7a87877ab78ba35292ef: Status 404 returned error can't find the container with id 4de3b47b3ee24c4b18ed6ff54307868d4caf83a0107a7a87877ab78ba35292ef
Jul 09 17:27:43.682321 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:43.682290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz" event={"ID":"ed0df847-ea7c-4ed4-9f5e-67543b25a753","Type":"ContainerStarted","Data":"ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef"}
Jul 09 17:27:43.682476 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:43.682334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz" event={"ID":"ed0df847-ea7c-4ed4-9f5e-67543b25a753","Type":"ContainerStarted","Data":"4de3b47b3ee24c4b18ed6ff54307868d4caf83a0107a7a87877ab78ba35292ef"}
Jul 09 17:27:43.701526 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:43.701477    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz" podStartSLOduration=1.701458305 podStartE2EDuration="1.701458305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:27:43.699875635 +0000 UTC m=+2204.397047962" watchObservedRunningTime="2026-07-09 17:27:43.701458305 +0000 UTC m=+2204.398630630"
Jul 09 17:27:47.560631 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:47.560591    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:27:53.152397 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:53.152364    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:53.152835 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:53.152413    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:53.164948 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:53.164925    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:27:53.741139 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:27:53.741113    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:28:03.851513 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:03.851483    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:28:23.312170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:23.312097    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"]
Jul 09 17:28:24.053653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:24.053624    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"]
Jul 09 17:28:26.459584 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:26.459547    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:28:26.946701 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:26.946671    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"]
Jul 09 17:28:26.950651 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:26.950631    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:26.953059 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:26.953039    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abbe5\"/\"rate-test-model-0abbe5-kserve-self-signed-certs\""
Jul 09 17:28:26.958893 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:26.958873    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"]
Jul 09 17:28:27.041745 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.041706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-dshm\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.041930 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.041753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8plq\" (UniqueName: \"kubernetes.io/projected/c10c4a7f-fbba-47f2-ba93-40bbfd905293-kube-api-access-p8plq\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.041930 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.041860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-model-cache\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.041930 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.041924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tmp-dir\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.042060 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.041944    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-home\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.042060 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.041970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tls-certs\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.142899 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.142867    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-home\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.142907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tls-certs\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.142941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-dshm\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.142962    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8plq\" (UniqueName: \"kubernetes.io/projected/c10c4a7f-fbba-47f2-ba93-40bbfd905293-kube-api-access-p8plq\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.142993    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-model-cache\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143235 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.143043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tmp-dir\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143342 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.143318    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-home\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143405 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.143383    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tmp-dir\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.143455 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.143407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-model-cache\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.145738 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.145715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-dshm\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.145942 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.145925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tls-certs\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.150922 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.150900    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8plq\" (UniqueName: \"kubernetes.io/projected/c10c4a7f-fbba-47f2-ba93-40bbfd905293-kube-api-access-p8plq\") pod \"rate-test-model-0abbe5-kserve-7b556c5795-chmbg\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") " pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.260828 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.260738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:27.396164 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.396065    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"]
Jul 09 17:28:27.398895 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:28:27.398869    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc10c4a7f_fbba_47f2_ba93_40bbfd905293.slice/crio-4cc6d72b6c43068cd9e6a849a3050ad6ec89363622aaefad321ffde44b5d9c81 WatchSource:0}: Error finding container 4cc6d72b6c43068cd9e6a849a3050ad6ec89363622aaefad321ffde44b5d9c81: Status 404 returned error can't find the container with id 4cc6d72b6c43068cd9e6a849a3050ad6ec89363622aaefad321ffde44b5d9c81
Jul 09 17:28:27.886814 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.886767    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg" event={"ID":"c10c4a7f-fbba-47f2-ba93-40bbfd905293","Type":"ContainerStarted","Data":"1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80"}
Jul 09 17:28:27.887180 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.886827    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg" event={"ID":"c10c4a7f-fbba-47f2-ba93-40bbfd905293","Type":"ContainerStarted","Data":"4cc6d72b6c43068cd9e6a849a3050ad6ec89363622aaefad321ffde44b5d9c81"}
Jul 09 17:28:27.908039 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:27.907994    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg" podStartSLOduration=1.907979601 podStartE2EDuration="1.907979601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:28:27.906599862 +0000 UTC m=+2248.603772183" watchObservedRunningTime="2026-07-09 17:28:27.907979601 +0000 UTC m=+2248.605151929"
Jul 09 17:28:37.261346 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:37.261305    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:37.261955 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:37.261368    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:37.274266 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:37.274238    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:37.943091 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:37.943063    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:28:38.332673 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.332555    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx" podUID="538f2879-3b82-4079-991b-8b503b05588f" containerName="main" containerID="cri-o://d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b" gracePeriod=45
Jul 09 17:28:38.589997 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.589941    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:28:38.632652 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.632623    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-tmp-dir\") pod \"538f2879-3b82-4079-991b-8b503b05588f\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") "
Jul 09 17:28:38.632808 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.632659    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-model-cache\") pod \"538f2879-3b82-4079-991b-8b503b05588f\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") "
Jul 09 17:28:38.632808 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.632715    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf497\" (UniqueName: \"kubernetes.io/projected/538f2879-3b82-4079-991b-8b503b05588f-kube-api-access-xf497\") pod \"538f2879-3b82-4079-991b-8b503b05588f\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") "
Jul 09 17:28:38.632808 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.632742    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538f2879-3b82-4079-991b-8b503b05588f-tls-certs\") pod \"538f2879-3b82-4079-991b-8b503b05588f\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") "
Jul 09 17:28:38.632963 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.632923    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "538f2879-3b82-4079-991b-8b503b05588f" (UID: "538f2879-3b82-4079-991b-8b503b05588f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:38.633024 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.632964    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-home\") pod \"538f2879-3b82-4079-991b-8b503b05588f\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") "
Jul 09 17:28:38.633024 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.633001    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-dshm\") pod \"538f2879-3b82-4079-991b-8b503b05588f\" (UID: \"538f2879-3b82-4079-991b-8b503b05588f\") "
Jul 09 17:28:38.633121 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.633013    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-model-cache" (OuterVolumeSpecName: "model-cache") pod "538f2879-3b82-4079-991b-8b503b05588f" (UID: "538f2879-3b82-4079-991b-8b503b05588f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:38.633259 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.633220    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-home" (OuterVolumeSpecName: "home") pod "538f2879-3b82-4079-991b-8b503b05588f" (UID: "538f2879-3b82-4079-991b-8b503b05588f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:38.633552 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.633290    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:38.633552 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.633304    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:38.635038 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.635016    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538f2879-3b82-4079-991b-8b503b05588f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "538f2879-3b82-4079-991b-8b503b05588f" (UID: "538f2879-3b82-4079-991b-8b503b05588f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:28:38.635138 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.635089    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538f2879-3b82-4079-991b-8b503b05588f-kube-api-access-xf497" (OuterVolumeSpecName: "kube-api-access-xf497") pod "538f2879-3b82-4079-991b-8b503b05588f" (UID: "538f2879-3b82-4079-991b-8b503b05588f"). InnerVolumeSpecName "kube-api-access-xf497". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:28:38.635206 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.635192    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-dshm" (OuterVolumeSpecName: "dshm") pod "538f2879-3b82-4079-991b-8b503b05588f" (UID: "538f2879-3b82-4079-991b-8b503b05588f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:38.733977 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.733941    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf497\" (UniqueName: \"kubernetes.io/projected/538f2879-3b82-4079-991b-8b503b05588f-kube-api-access-xf497\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:38.733977 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.733969    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538f2879-3b82-4079-991b-8b503b05588f-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:38.733977 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.733980    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:38.734184 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.733988    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538f2879-3b82-4079-991b-8b503b05588f-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:38.936603 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.936573    2567 generic.go:356] "Generic (PLEG): container finished" podID="538f2879-3b82-4079-991b-8b503b05588f" containerID="d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b" exitCode=0
Jul 09 17:28:38.936786 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.936608    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"
Jul 09 17:28:38.936786 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.936654    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx" event={"ID":"538f2879-3b82-4079-991b-8b503b05588f","Type":"ContainerDied","Data":"d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b"}
Jul 09 17:28:38.936786 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.936695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx" event={"ID":"538f2879-3b82-4079-991b-8b503b05588f","Type":"ContainerDied","Data":"4b0fa7c469f11d018ddb23c4c3aa012382965da0dd62faf03bbe380db2554a1e"}
Jul 09 17:28:38.936786 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.936713    2567 scope.go:117] "RemoveContainer" containerID="d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b"
Jul 09 17:28:38.951976 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.951954    2567 scope.go:117] "RemoveContainer" containerID="d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b"
Jul 09 17:28:38.952237 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:28:38.952217    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b\": container with ID starting with d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b not found: ID does not exist" containerID="d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b"
Jul 09 17:28:38.952311 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.952245    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b"} err="failed to get container status \"d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b\": rpc error: code = NotFound desc = could not find container \"d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b\": container with ID starting with d7281edb5d2d3bacc09264a88035bc9cb26a32badb55af57c0173c90bd103a9b not found: ID does not exist"
Jul 09 17:28:38.962852 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.962829    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"]
Jul 09 17:28:38.968597 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:38.968575    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/sub-test-model-0abbe5-kserve-7bddf86f85-72crx"]
Jul 09 17:28:39.075234 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.075205    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz" podUID="ed0df847-ea7c-4ed4-9f5e-67543b25a753" containerName="main" containerID="cri-o://ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef" gracePeriod=45
Jul 09 17:28:39.314275 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.314252    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:28:39.339144 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339119    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-home\") pod \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") "
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339170    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-dshm\") pod \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") "
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339202    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-model-cache\") pod \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") "
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339237    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp942\" (UniqueName: \"kubernetes.io/projected/ed0df847-ea7c-4ed4-9f5e-67543b25a753-kube-api-access-pp942\") pod \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") "
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339305    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tmp-dir\") pod \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") "
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339362    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tls-certs\") pod \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\" (UID: \"ed0df847-ea7c-4ed4-9f5e-67543b25a753\") "
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339364    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-home" (OuterVolumeSpecName: "home") pod "ed0df847-ea7c-4ed4-9f5e-67543b25a753" (UID: "ed0df847-ea7c-4ed4-9f5e-67543b25a753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:39.339532 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed0df847-ea7c-4ed4-9f5e-67543b25a753" (UID: "ed0df847-ea7c-4ed4-9f5e-67543b25a753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:39.340115 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339621    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:39.340115 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339639    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:39.340115 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.339877    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed0df847-ea7c-4ed4-9f5e-67543b25a753" (UID: "ed0df847-ea7c-4ed4-9f5e-67543b25a753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:39.341586 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.341558    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-dshm" (OuterVolumeSpecName: "dshm") pod "ed0df847-ea7c-4ed4-9f5e-67543b25a753" (UID: "ed0df847-ea7c-4ed4-9f5e-67543b25a753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:39.341693 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.341600    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed0df847-ea7c-4ed4-9f5e-67543b25a753" (UID: "ed0df847-ea7c-4ed4-9f5e-67543b25a753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:28:39.342108 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.342086    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0df847-ea7c-4ed4-9f5e-67543b25a753-kube-api-access-pp942" (OuterVolumeSpecName: "kube-api-access-pp942") pod "ed0df847-ea7c-4ed4-9f5e-67543b25a753" (UID: "ed0df847-ea7c-4ed4-9f5e-67543b25a753"). InnerVolumeSpecName "kube-api-access-pp942". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:28:39.440569 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.440535    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp942\" (UniqueName: \"kubernetes.io/projected/ed0df847-ea7c-4ed4-9f5e-67543b25a753-kube-api-access-pp942\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:39.440569 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.440564    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0df847-ea7c-4ed4-9f5e-67543b25a753-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:39.440569 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.440573    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:39.440824 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.440582    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0df847-ea7c-4ed4-9f5e-67543b25a753-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:28:39.872320 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.872286    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="538f2879-3b82-4079-991b-8b503b05588f" path="/var/lib/kubelet/pods/538f2879-3b82-4079-991b-8b503b05588f/volumes"
Jul 09 17:28:39.943946 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.943909    2567 generic.go:356] "Generic (PLEG): container finished" podID="ed0df847-ea7c-4ed4-9f5e-67543b25a753" containerID="ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef" exitCode=0
Jul 09 17:28:39.944139 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.943959    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"
Jul 09 17:28:39.944139 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.944004    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz" event={"ID":"ed0df847-ea7c-4ed4-9f5e-67543b25a753","Type":"ContainerDied","Data":"ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef"}
Jul 09 17:28:39.944139 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.944058    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz" event={"ID":"ed0df847-ea7c-4ed4-9f5e-67543b25a753","Type":"ContainerDied","Data":"4de3b47b3ee24c4b18ed6ff54307868d4caf83a0107a7a87877ab78ba35292ef"}
Jul 09 17:28:39.944139 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.944082    2567 scope.go:117] "RemoveContainer" containerID="ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef"
Jul 09 17:28:39.954676 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.954656    2567 scope.go:117] "RemoveContainer" containerID="ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef"
Jul 09 17:28:39.954961 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:28:39.954942    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef\": container with ID starting with ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef not found: ID does not exist" containerID="ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef"
Jul 09 17:28:39.955039 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.954969    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef"} err="failed to get container status \"ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef\": rpc error: code = NotFound desc = could not find container \"ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef\": container with ID starting with ef07c10beca036c65da3c32f72e6391acd4b6587cef89850359050194cbfa7ef not found: ID does not exist"
Jul 09 17:28:39.960231 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.960158    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"]
Jul 09 17:28:39.962108 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:39.962088    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/sub-test-model-265f70-kserve-84ccdfdccf-24phz"]
Jul 09 17:28:41.872784 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:41.872748    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed0df847-ea7c-4ed4-9f5e-67543b25a753" path="/var/lib/kubelet/pods/ed0df847-ea7c-4ed4-9f5e-67543b25a753/volumes"
Jul 09 17:28:43.659294 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:43.659251    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:28:58.958483 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958454    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"]
Jul 09 17:28:58.958878 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958828    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538f2879-3b82-4079-991b-8b503b05588f" containerName="main"
Jul 09 17:28:58.958878 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958839    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="538f2879-3b82-4079-991b-8b503b05588f" containerName="main"
Jul 09 17:28:58.958878 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958859    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0df847-ea7c-4ed4-9f5e-67543b25a753" containerName="main"
Jul 09 17:28:58.958878 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958864    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0df847-ea7c-4ed4-9f5e-67543b25a753" containerName="main"
Jul 09 17:28:58.959030 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958931    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="538f2879-3b82-4079-991b-8b503b05588f" containerName="main"
Jul 09 17:28:58.959030 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.958949    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0df847-ea7c-4ed4-9f5e-67543b25a753" containerName="main"
Jul 09 17:28:58.961930 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.961913    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:58.965902 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.965880    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"default-dockercfg-rctq9\""
Jul 09 17:28:58.966022 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.965907    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"openshift-service-ca.crt\""
Jul 09 17:28:58.966022 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.965974    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"rate-test-model-265f70-kserve-self-signed-certs\""
Jul 09 17:28:58.966022 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.965980    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-265f70\"/\"kube-root-ca.crt\""
Jul 09 17:28:58.975046 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:58.973790    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"]
Jul 09 17:28:59.008673 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.008644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tls-certs\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.008860 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.008717    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-model-cache\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.008860 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.008752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tmp-dir\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.008860 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.008791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfzm\" (UniqueName: \"kubernetes.io/projected/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-kube-api-access-xpfzm\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.009097 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.008882    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-home\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.009097 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.008911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-dshm\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.109544 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tls-certs\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.109653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109565    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-model-cache\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.109653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tmp-dir\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.109653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfzm\" (UniqueName: \"kubernetes.io/projected/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-kube-api-access-xpfzm\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.109653 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-home\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.109889 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-dshm\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.110023 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.109998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-model-cache\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.110110 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.110016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tmp-dir\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.110170 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.110136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-home\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.111988 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.111969    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-dshm\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.112312 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.112293    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tls-certs\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.117162 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.117137    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfzm\" (UniqueName: \"kubernetes.io/projected/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-kube-api-access-xpfzm\") pod \"rate-test-model-265f70-kserve-ccb4ddcb7-5km7j\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") " pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.271020 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.270929    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:28:59.403244 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:28:59.403221    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"]
Jul 09 17:28:59.405745 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:28:59.405709    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e6ddf37_762b_4fe7_a01b_1c2ee4c0a48e.slice/crio-6f8a5c242e9c919b055902145398a19a4526f9764f9e7da715caec87b68bf77d WatchSource:0}: Error finding container 6f8a5c242e9c919b055902145398a19a4526f9764f9e7da715caec87b68bf77d: Status 404 returned error can't find the container with id 6f8a5c242e9c919b055902145398a19a4526f9764f9e7da715caec87b68bf77d
Jul 09 17:29:00.041447 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:00.041405    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j" event={"ID":"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e","Type":"ContainerStarted","Data":"4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433"}
Jul 09 17:29:00.041447 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:00.041450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j" event={"ID":"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e","Type":"ContainerStarted","Data":"6f8a5c242e9c919b055902145398a19a4526f9764f9e7da715caec87b68bf77d"}
Jul 09 17:29:00.058774 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:00.058727    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j" podStartSLOduration=2.058710006 podStartE2EDuration="2.058710006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:29:00.057274113 +0000 UTC m=+2280.754446441" watchObservedRunningTime="2026-07-09 17:29:00.058710006 +0000 UTC m=+2280.755882332"
Jul 09 17:29:03.366436 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:03.366394    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:29:09.271388 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:09.271349    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:29:09.272040 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:09.271411    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:29:09.283612 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:09.283580    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:29:10.103428 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:10.103401    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:29:47.458140 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:29:47.458099    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:30:10.461855 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:10.461816    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:30:14.981494 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:14.981458    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"]
Jul 09 17:30:15.750753 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:15.750723    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"]
Jul 09 17:30:27.455606 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:27.455572    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:30:30.003552 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.003497    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg" podUID="c10c4a7f-fbba-47f2-ba93-40bbfd905293" containerName="main" containerID="cri-o://1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80" gracePeriod=45
Jul 09 17:30:30.248105 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.248075    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:30:30.363123 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363101    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-model-cache\") pod \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") "
Jul 09 17:30:30.363296 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363141    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8plq\" (UniqueName: \"kubernetes.io/projected/c10c4a7f-fbba-47f2-ba93-40bbfd905293-kube-api-access-p8plq\") pod \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") "
Jul 09 17:30:30.363296 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363194    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-dshm\") pod \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") "
Jul 09 17:30:30.363296 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363218    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-home\") pod \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") "
Jul 09 17:30:30.363296 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363240    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tls-certs\") pod \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") "
Jul 09 17:30:30.363509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363340    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tmp-dir\") pod \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\" (UID: \"c10c4a7f-fbba-47f2-ba93-40bbfd905293\") "
Jul 09 17:30:30.363509 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363413    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-model-cache" (OuterVolumeSpecName: "model-cache") pod "c10c4a7f-fbba-47f2-ba93-40bbfd905293" (UID: "c10c4a7f-fbba-47f2-ba93-40bbfd905293"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:30.363616 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-home" (OuterVolumeSpecName: "home") pod "c10c4a7f-fbba-47f2-ba93-40bbfd905293" (UID: "c10c4a7f-fbba-47f2-ba93-40bbfd905293"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:30.363672 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363615    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c10c4a7f-fbba-47f2-ba93-40bbfd905293" (UID: "c10c4a7f-fbba-47f2-ba93-40bbfd905293"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:30.363735 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363697    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:30.363735 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363726    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:30.363879 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.363740    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:30.365666 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.365625    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-dshm" (OuterVolumeSpecName: "dshm") pod "c10c4a7f-fbba-47f2-ba93-40bbfd905293" (UID: "c10c4a7f-fbba-47f2-ba93-40bbfd905293"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:30.365965 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.365937    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10c4a7f-fbba-47f2-ba93-40bbfd905293-kube-api-access-p8plq" (OuterVolumeSpecName: "kube-api-access-p8plq") pod "c10c4a7f-fbba-47f2-ba93-40bbfd905293" (UID: "c10c4a7f-fbba-47f2-ba93-40bbfd905293"). InnerVolumeSpecName "kube-api-access-p8plq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:30.366076 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.365998    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c10c4a7f-fbba-47f2-ba93-40bbfd905293" (UID: "c10c4a7f-fbba-47f2-ba93-40bbfd905293"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:30.446412 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.446385    2567 generic.go:356] "Generic (PLEG): container finished" podID="c10c4a7f-fbba-47f2-ba93-40bbfd905293" containerID="1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80" exitCode=0
Jul 09 17:30:30.446511 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.446432    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"
Jul 09 17:30:30.446511 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.446436    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg" event={"ID":"c10c4a7f-fbba-47f2-ba93-40bbfd905293","Type":"ContainerDied","Data":"1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80"}
Jul 09 17:30:30.446511 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.446474    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg" event={"ID":"c10c4a7f-fbba-47f2-ba93-40bbfd905293","Type":"ContainerDied","Data":"4cc6d72b6c43068cd9e6a849a3050ad6ec89363622aaefad321ffde44b5d9c81"}
Jul 09 17:30:30.446511 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.446491    2567 scope.go:117] "RemoveContainer" containerID="1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80"
Jul 09 17:30:30.454899 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.454873    2567 scope.go:117] "RemoveContainer" containerID="1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80"
Jul 09 17:30:30.455149 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:30.455131    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80\": container with ID starting with 1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80 not found: ID does not exist" containerID="1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80"
Jul 09 17:30:30.455199 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.455157    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80"} err="failed to get container status \"1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80\": rpc error: code = NotFound desc = could not find container \"1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80\": container with ID starting with 1ff224d5ab0feb4aaa6359ce014804de8ab7572396cbde816037914bd9ac3f80 not found: ID does not exist"
Jul 09 17:30:30.464879 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.464846    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8plq\" (UniqueName: \"kubernetes.io/projected/c10c4a7f-fbba-47f2-ba93-40bbfd905293-kube-api-access-p8plq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:30.464966 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.464882    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10c4a7f-fbba-47f2-ba93-40bbfd905293-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:30.464966 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.464897    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10c4a7f-fbba-47f2-ba93-40bbfd905293-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:30.467406 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.467383    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"]
Jul 09 17:30:30.471348 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.471329    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0abbe5/rate-test-model-0abbe5-kserve-7b556c5795-chmbg"]
Jul 09 17:30:30.771507 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:30.771403    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j" podUID="1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" containerName="main" containerID="cri-o://4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433" gracePeriod=45
Jul 09 17:30:31.010206 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.010182    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:30:31.070108 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070052    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpfzm\" (UniqueName: \"kubernetes.io/projected/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-kube-api-access-xpfzm\") pod \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") "
Jul 09 17:30:31.070229 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070110    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-home\") pod \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") "
Jul 09 17:30:31.070229 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070177    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tmp-dir\") pod \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") "
Jul 09 17:30:31.070229 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070205    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-model-cache\") pod \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") "
Jul 09 17:30:31.070229 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070225    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-dshm\") pod \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") "
Jul 09 17:30:31.070431 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070255    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tls-certs\") pod \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\" (UID: \"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e\") "
Jul 09 17:30:31.070431 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070378    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-home" (OuterVolumeSpecName: "home") pod "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" (UID: "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:31.070431 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070398    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" (UID: "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:31.070538 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070478    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" (UID: "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:31.070600 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070578    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.070600 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070603    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.070742 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.070612    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.072267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.072241    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-kube-api-access-xpfzm" (OuterVolumeSpecName: "kube-api-access-xpfzm") pod "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" (UID: "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e"). InnerVolumeSpecName "kube-api-access-xpfzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:31.072351 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.072325    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" (UID: "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:31.072485 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.072463    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-dshm" (OuterVolumeSpecName: "dshm") pod "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" (UID: "1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:31.171204 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.171183    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.171204 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.171202    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.171322 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.171212    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfzm\" (UniqueName: \"kubernetes.io/projected/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e-kube-api-access-xpfzm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.452095 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.452065    2567 generic.go:356] "Generic (PLEG): container finished" podID="1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" containerID="4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433" exitCode=0
Jul 09 17:30:31.452241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.452111    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"
Jul 09 17:30:31.452241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.452147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j" event={"ID":"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e","Type":"ContainerDied","Data":"4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433"}
Jul 09 17:30:31.452241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.452185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j" event={"ID":"1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e","Type":"ContainerDied","Data":"6f8a5c242e9c919b055902145398a19a4526f9764f9e7da715caec87b68bf77d"}
Jul 09 17:30:31.452241 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.452208    2567 scope.go:117] "RemoveContainer" containerID="4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433"
Jul 09 17:30:31.461182 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.461164    2567 scope.go:117] "RemoveContainer" containerID="4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433"
Jul 09 17:30:31.461466 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:31.461450    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433\": container with ID starting with 4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433 not found: ID does not exist" containerID="4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433"
Jul 09 17:30:31.461537 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.461475    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433"} err="failed to get container status \"4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433\": rpc error: code = NotFound desc = could not find container \"4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433\": container with ID starting with 4094c9e105b103ca10b927f9bea060d0e5402426d9356e19db211e57001b6433 not found: ID does not exist"
Jul 09 17:30:31.472165 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.472141    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"]
Jul 09 17:30:31.475895 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.475875    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-265f70/rate-test-model-265f70-kserve-ccb4ddcb7-5km7j"]
Jul 09 17:30:31.872267 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.872240    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" path="/var/lib/kubelet/pods/1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e/volumes"
Jul 09 17:30:31.872622 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:31.872610    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10c4a7f-fbba-47f2-ba93-40bbfd905293" path="/var/lib/kubelet/pods/c10c4a7f-fbba-47f2-ba93-40bbfd905293/volumes"
Jul 09 17:30:32.061780 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.061754    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7561-140577980377968"]
Jul 09 17:30:32.062104 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.062087    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" containerName="main"
Jul 09 17:30:32.062104 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.062098    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" containerName="main"
Jul 09 17:30:32.062187 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.062106    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10c4a7f-fbba-47f2-ba93-40bbfd905293" containerName="main"
Jul 09 17:30:32.062187 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.062111    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10c4a7f-fbba-47f2-ba93-40bbfd905293" containerName="main"
Jul 09 17:30:32.062187 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.062173    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6ddf37-762b-4fe7-a01b-1c2ee4c0a48e" containerName="main"
Jul 09 17:30:32.062187 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.062182    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10c4a7f-fbba-47f2-ba93-40bbfd905293" containerName="main"
Jul 09 17:30:32.066443 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.066426    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:32.069037 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.068998    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-4c6xj\""
Jul 09 17:30:32.071166 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.071141    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7561-140577980377968"]
Jul 09 17:30:32.179908 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.179847    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwjq\" (UniqueName: \"kubernetes.io/projected/03100a0e-fef9-4b98-92f0-f7bc21327dce-kube-api-access-dhwjq\") pod \"test-curl-7561-140577980377968\" (UID: \"03100a0e-fef9-4b98-92f0-f7bc21327dce\") " pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:32.280211 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.280183    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwjq\" (UniqueName: \"kubernetes.io/projected/03100a0e-fef9-4b98-92f0-f7bc21327dce-kube-api-access-dhwjq\") pod \"test-curl-7561-140577980377968\" (UID: \"03100a0e-fef9-4b98-92f0-f7bc21327dce\") " pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:32.288203 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.288184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwjq\" (UniqueName: \"kubernetes.io/projected/03100a0e-fef9-4b98-92f0-f7bc21327dce-kube-api-access-dhwjq\") pod \"test-curl-7561-140577980377968\" (UID: \"03100a0e-fef9-4b98-92f0-f7bc21327dce\") " pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:32.376196 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.376178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:32.507975 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.507948    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7561-140577980377968"]
Jul 09 17:30:32.511288 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:30:32.511257    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03100a0e_fef9_4b98_92f0_f7bc21327dce.slice/crio-8cf060110e7cfb62ce3487bb4e5061d9977c8f02065eaf29ee629a4c1dfa68d1 WatchSource:0}: Error finding container 8cf060110e7cfb62ce3487bb4e5061d9977c8f02065eaf29ee629a4c1dfa68d1: Status 404 returned error can't find the container with id 8cf060110e7cfb62ce3487bb4e5061d9977c8f02065eaf29ee629a4c1dfa68d1
Jul 09 17:30:32.512613 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:32.512596    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:30:33.465671 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:33.465640    2567 generic.go:356] "Generic (PLEG): container finished" podID="03100a0e-fef9-4b98-92f0-f7bc21327dce" containerID="7e42bec6ee68fd648b0e58197479539ac17ba5e40d3ebc0b78b04d0915e047f4" exitCode=0
Jul 09 17:30:33.466001 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:33.465721    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7561-140577980377968" event={"ID":"03100a0e-fef9-4b98-92f0-f7bc21327dce","Type":"ContainerDied","Data":"7e42bec6ee68fd648b0e58197479539ac17ba5e40d3ebc0b78b04d0915e047f4"}
Jul 09 17:30:33.466001 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:33.465762    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7561-140577980377968" event={"ID":"03100a0e-fef9-4b98-92f0-f7bc21327dce","Type":"ContainerStarted","Data":"8cf060110e7cfb62ce3487bb4e5061d9977c8f02065eaf29ee629a4c1dfa68d1"}
Jul 09 17:30:34.611274 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.611254    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:34.646667 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.646645    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7561-140577980377968_03100a0e-fef9-4b98-92f0-f7bc21327dce/test-curl-7561-140577980377968/0.log"
Jul 09 17:30:34.669837 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.669815    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7561-140577980377968"]
Jul 09 17:30:34.671738 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.671703    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7561-140577980377968"]
Jul 09 17:30:34.701492 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.701469    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhwjq\" (UniqueName: \"kubernetes.io/projected/03100a0e-fef9-4b98-92f0-f7bc21327dce-kube-api-access-dhwjq\") pod \"03100a0e-fef9-4b98-92f0-f7bc21327dce\" (UID: \"03100a0e-fef9-4b98-92f0-f7bc21327dce\") "
Jul 09 17:30:34.703643 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.703623    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03100a0e-fef9-4b98-92f0-f7bc21327dce-kube-api-access-dhwjq" (OuterVolumeSpecName: "kube-api-access-dhwjq") pod "03100a0e-fef9-4b98-92f0-f7bc21327dce" (UID: "03100a0e-fef9-4b98-92f0-f7bc21327dce"). InnerVolumeSpecName "kube-api-access-dhwjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:34.802349 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:34.802297    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwjq\" (UniqueName: \"kubernetes.io/projected/03100a0e-fef9-4b98-92f0-f7bc21327dce-kube-api-access-dhwjq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:35.476117 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.476092    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf060110e7cfb62ce3487bb4e5061d9977c8f02065eaf29ee629a4c1dfa68d1"
Jul 09 17:30:35.476117 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.476107    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:35.485419 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:35.485391    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7561-140577980377968\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:35.550252 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.550230    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"]
Jul 09 17:30:35.550451 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.550423    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5" podUID="4be88f08-ccf7-4ef2-97a1-239925442c3f" containerName="maas-api" containerID="cri-o://15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea" gracePeriod=30
Jul 09 17:30:35.569864 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:35.569833    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7561-140577980377968\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:35.791322 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.791290    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:30:35.810265 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:35.810232    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7561-140577980377968\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" pod="openshift-ingress/test-curl-7561-140577980377968"
Jul 09 17:30:35.871827 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.871787    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" path="/var/lib/kubelet/pods/03100a0e-fef9-4b98-92f0-f7bc21327dce/volumes"
Jul 09 17:30:35.910890 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.910865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be88f08-ccf7-4ef2-97a1-239925442c3f-openshift-service-ca\") pod \"4be88f08-ccf7-4ef2-97a1-239925442c3f\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") "
Jul 09 17:30:35.911041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.910930    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqjgs\" (UniqueName: \"kubernetes.io/projected/4be88f08-ccf7-4ef2-97a1-239925442c3f-kube-api-access-qqjgs\") pod \"4be88f08-ccf7-4ef2-97a1-239925442c3f\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") "
Jul 09 17:30:35.911041 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.910992    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be88f08-ccf7-4ef2-97a1-239925442c3f-maas-api-tls\") pod \"4be88f08-ccf7-4ef2-97a1-239925442c3f\" (UID: \"4be88f08-ccf7-4ef2-97a1-239925442c3f\") "
Jul 09 17:30:35.911282 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.911258    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be88f08-ccf7-4ef2-97a1-239925442c3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4be88f08-ccf7-4ef2-97a1-239925442c3f" (UID: "4be88f08-ccf7-4ef2-97a1-239925442c3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:30:35.913126 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.913102    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be88f08-ccf7-4ef2-97a1-239925442c3f-kube-api-access-qqjgs" (OuterVolumeSpecName: "kube-api-access-qqjgs") pod "4be88f08-ccf7-4ef2-97a1-239925442c3f" (UID: "4be88f08-ccf7-4ef2-97a1-239925442c3f"). InnerVolumeSpecName "kube-api-access-qqjgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:35.913212 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:35.913142    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be88f08-ccf7-4ef2-97a1-239925442c3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4be88f08-ccf7-4ef2-97a1-239925442c3f" (UID: "4be88f08-ccf7-4ef2-97a1-239925442c3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:36.012253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.012229    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be88f08-ccf7-4ef2-97a1-239925442c3f-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:36.012253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.012249    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjgs\" (UniqueName: \"kubernetes.io/projected/4be88f08-ccf7-4ef2-97a1-239925442c3f-kube-api-access-qqjgs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:36.012378 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.012258    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4be88f08-ccf7-4ef2-97a1-239925442c3f-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:36.481776 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.481745    2567 generic.go:356] "Generic (PLEG): container finished" podID="4be88f08-ccf7-4ef2-97a1-239925442c3f" containerID="15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea" exitCode=0
Jul 09 17:30:36.481948 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.481786    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"
Jul 09 17:30:36.481948 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.481829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5" event={"ID":"4be88f08-ccf7-4ef2-97a1-239925442c3f","Type":"ContainerDied","Data":"15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea"}
Jul 09 17:30:36.481948 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.481888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5" event={"ID":"4be88f08-ccf7-4ef2-97a1-239925442c3f","Type":"ContainerDied","Data":"e5a2b2070499b9e47056363b7f8d02b43a12b7029b0be7ac35d1b9666cae1902"}
Jul 09 17:30:36.481948 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.481913    2567 scope.go:117] "RemoveContainer" containerID="15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea"
Jul 09 17:30:36.491179 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.491164    2567 scope.go:117] "RemoveContainer" containerID="15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea"
Jul 09 17:30:36.491433 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:36.491416    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea\": container with ID starting with 15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea not found: ID does not exist" containerID="15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea"
Jul 09 17:30:36.491476 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.491441    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea"} err="failed to get container status \"15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea\": rpc error: code = NotFound desc = could not find container \"15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea\": container with ID starting with 15ee00d257395118532bf2e7145c2b68502dce107002566043cf64cd17c495ea not found: ID does not exist"
Jul 09 17:30:36.503305 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.503281    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"]
Jul 09 17:30:36.506471 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:36.506450    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0abbe5-598b4899f5-nflg5"]
Jul 09 17:30:37.871695 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:37.871660    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be88f08-ccf7-4ef2-97a1-239925442c3f" path="/var/lib/kubelet/pods/4be88f08-ccf7-4ef2-97a1-239925442c3f/volumes"
Jul 09 17:30:40.864063 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:40.864031    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:30:42.788302 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:42.788268    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"]
Jul 09 17:30:42.788777 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:42.788497    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k" podUID="0b186b48-a909-4fe7-985c-03337caff987" containerName="maas-api" containerID="cri-o://ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7" gracePeriod=30
Jul 09 17:30:43.028501 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.028475    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:30:43.068722 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.068663    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b186b48-a909-4fe7-985c-03337caff987-maas-api-tls\") pod \"0b186b48-a909-4fe7-985c-03337caff987\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") "
Jul 09 17:30:43.068722 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.068692    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmj96\" (UniqueName: \"kubernetes.io/projected/0b186b48-a909-4fe7-985c-03337caff987-kube-api-access-cmj96\") pod \"0b186b48-a909-4fe7-985c-03337caff987\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") "
Jul 09 17:30:43.068920 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.068773    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b186b48-a909-4fe7-985c-03337caff987-openshift-service-ca\") pod \"0b186b48-a909-4fe7-985c-03337caff987\" (UID: \"0b186b48-a909-4fe7-985c-03337caff987\") "
Jul 09 17:30:43.069159 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.069135    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b186b48-a909-4fe7-985c-03337caff987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b186b48-a909-4fe7-985c-03337caff987" (UID: "0b186b48-a909-4fe7-985c-03337caff987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:30:43.070880 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.070859    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b186b48-a909-4fe7-985c-03337caff987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b186b48-a909-4fe7-985c-03337caff987" (UID: "0b186b48-a909-4fe7-985c-03337caff987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:43.070966 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.070944    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b186b48-a909-4fe7-985c-03337caff987-kube-api-access-cmj96" (OuterVolumeSpecName: "kube-api-access-cmj96") pod "0b186b48-a909-4fe7-985c-03337caff987" (UID: "0b186b48-a909-4fe7-985c-03337caff987"). InnerVolumeSpecName "kube-api-access-cmj96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:43.169713 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.169688    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b186b48-a909-4fe7-985c-03337caff987-openshift-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:43.169713 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.169710    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b186b48-a909-4fe7-985c-03337caff987-maas-api-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:43.169872 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.169719    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmj96\" (UniqueName: \"kubernetes.io/projected/0b186b48-a909-4fe7-985c-03337caff987-kube-api-access-cmj96\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Jul 09 17:30:43.514385 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.514354    2567 generic.go:356] "Generic (PLEG): container finished" podID="0b186b48-a909-4fe7-985c-03337caff987" containerID="ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7" exitCode=0
Jul 09 17:30:43.514550 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.514399    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"
Jul 09 17:30:43.514550 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.514446    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k" event={"ID":"0b186b48-a909-4fe7-985c-03337caff987","Type":"ContainerDied","Data":"ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7"}
Jul 09 17:30:43.514550 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.514508    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k" event={"ID":"0b186b48-a909-4fe7-985c-03337caff987","Type":"ContainerDied","Data":"6ae2e0f6fa5a934e2b5c047e292d406017d14ecbd6bd96a955822b7aea066c1b"}
Jul 09 17:30:43.514550 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.514537    2567 scope.go:117] "RemoveContainer" containerID="ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7"
Jul 09 17:30:43.523680 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.523664    2567 scope.go:117] "RemoveContainer" containerID="ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7"
Jul 09 17:30:43.523954 ip-10-0-128-131 kubenswrapper[2567]: E0709 17:30:43.523934    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7\": container with ID starting with ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7 not found: ID does not exist" containerID="ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7"
Jul 09 17:30:43.524030 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.523962    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7"} err="failed to get container status \"ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7\": rpc error: code = NotFound desc = could not find container \"ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7\": container with ID starting with ba01aec89da5f8cf9c027117b3e2484d98397b36799d0dfaf17eacf6676990c7 not found: ID does not exist"
Jul 09 17:30:43.534629 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.534609    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"]
Jul 09 17:30:43.540414 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.540397    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-265f70-8568b9cb49-rgn2k"]
Jul 09 17:30:43.873070 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:43.873042    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b186b48-a909-4fe7-985c-03337caff987" path="/var/lib/kubelet/pods/0b186b48-a909-4fe7-985c-03337caff987/volumes"
Jul 09 17:30:49.261843 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:49.261777    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f57fdb9f-hw4v8_060e6d2b-1b47-42b4-a547-953cacc3b41f/authorino/0.log"
Jul 09 17:30:51.062148 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:51.062118    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcks8"]
Jul 09 17:30:54.066859 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:54.066826    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bjvcf_3f1d1892-8663-4908-b974-f70fc205b18f/manager/0.log"
Jul 09 17:30:54.189684 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:54.189653    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mvspw_8d834298-f377-43c7-ade1-e5e12c818d78/manager/0.log"
Jul 09 17:30:54.937897 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:54.937869    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789dddf866-mnvbm_e71bbb6e-28e9-490c-b09b-0d5256f061bf/copy-manifests/0.log"
Jul 09 17:30:54.944668 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:54.944648    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789dddf866-mnvbm_e71bbb6e-28e9-490c-b09b-0d5256f061bf/manager/0.log"
Jul 09 17:30:57.039152 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:57.039119    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f57fdb9f-hw4v8_060e6d2b-1b47-42b4-a547-953cacc3b41f/authorino/0.log"
Jul 09 17:30:57.381217 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:57.381194    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4s49b_0209346f-8206-4716-a6bf-d1631786923c/kuadrant-console-plugin/0.log"
Jul 09 17:30:57.771055 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:57.770962    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fcks8_6c9bed73-2881-4ba0-bc0f-d759bab1e883/limitador/0.log"
Jul 09 17:30:58.596476 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:30:58.596443    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cc96589f5-jcvjk_7108effc-c7bd-4c82-9a09-c3fa30ef8102/kube-auth-proxy/0.log"
Jul 09 17:31:00.021816 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:00.021754    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-mhgfg_c2257769-7410-4250-b057-be64c8bb78d2/main/0.log"
Jul 09 17:31:04.161055 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161019    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qjspn/must-gather-jrpcl"]
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161345    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" containerName="test-curl-7561-140577980377968"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161356    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" containerName="test-curl-7561-140577980377968"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161371    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be88f08-ccf7-4ef2-97a1-239925442c3f" containerName="maas-api"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161377    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be88f08-ccf7-4ef2-97a1-239925442c3f" containerName="maas-api"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161384    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b186b48-a909-4fe7-985c-03337caff987" containerName="maas-api"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161389    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b186b48-a909-4fe7-985c-03337caff987" containerName="maas-api"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161449    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b186b48-a909-4fe7-985c-03337caff987" containerName="maas-api"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161457    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be88f08-ccf7-4ef2-97a1-239925442c3f" containerName="maas-api"
Jul 09 17:31:04.161500 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.161465    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="03100a0e-fef9-4b98-92f0-f7bc21327dce" containerName="test-curl-7561-140577980377968"
Jul 09 17:31:04.164458 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.164437    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.167126 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.167103    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qjspn\"/\"kube-root-ca.crt\""
Jul 09 17:31:04.167253 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.167192    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qjspn\"/\"openshift-service-ca.crt\""
Jul 09 17:31:04.167330 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.167248    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qjspn\"/\"default-dockercfg-d452k\""
Jul 09 17:31:04.175782 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.175759    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjspn/must-gather-jrpcl"]
Jul 09 17:31:04.249184 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.249148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06f90a3a-c2c7-4b42-a781-1559fde2ae76-must-gather-output\") pod \"must-gather-jrpcl\" (UID: \"06f90a3a-c2c7-4b42-a781-1559fde2ae76\") " pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.249365 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.249210    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xbx\" (UniqueName: \"kubernetes.io/projected/06f90a3a-c2c7-4b42-a781-1559fde2ae76-kube-api-access-w5xbx\") pod \"must-gather-jrpcl\" (UID: \"06f90a3a-c2c7-4b42-a781-1559fde2ae76\") " pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.350638 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.350603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xbx\" (UniqueName: \"kubernetes.io/projected/06f90a3a-c2c7-4b42-a781-1559fde2ae76-kube-api-access-w5xbx\") pod \"must-gather-jrpcl\" (UID: \"06f90a3a-c2c7-4b42-a781-1559fde2ae76\") " pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.350849 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.350697    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06f90a3a-c2c7-4b42-a781-1559fde2ae76-must-gather-output\") pod \"must-gather-jrpcl\" (UID: \"06f90a3a-c2c7-4b42-a781-1559fde2ae76\") " pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.351084 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.351065    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06f90a3a-c2c7-4b42-a781-1559fde2ae76-must-gather-output\") pod \"must-gather-jrpcl\" (UID: \"06f90a3a-c2c7-4b42-a781-1559fde2ae76\") " pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.359568 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.359537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xbx\" (UniqueName: \"kubernetes.io/projected/06f90a3a-c2c7-4b42-a781-1559fde2ae76-kube-api-access-w5xbx\") pod \"must-gather-jrpcl\" (UID: \"06f90a3a-c2c7-4b42-a781-1559fde2ae76\") " pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.474279 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.474192    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjspn/must-gather-jrpcl"
Jul 09 17:31:04.607300 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.607222    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjspn/must-gather-jrpcl"]
Jul 09 17:31:04.609623 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:31:04.609594    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f90a3a_c2c7_4b42_a781_1559fde2ae76.slice/crio-8a4483c33ce00b63d519f6970e6d8b15794440b85db645885b6fd8bcbe16c804 WatchSource:0}: Error finding container 8a4483c33ce00b63d519f6970e6d8b15794440b85db645885b6fd8bcbe16c804: Status 404 returned error can't find the container with id 8a4483c33ce00b63d519f6970e6d8b15794440b85db645885b6fd8bcbe16c804
Jul 09 17:31:04.620640 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:04.620604    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjspn/must-gather-jrpcl" event={"ID":"06f90a3a-c2c7-4b42-a781-1559fde2ae76","Type":"ContainerStarted","Data":"8a4483c33ce00b63d519f6970e6d8b15794440b85db645885b6fd8bcbe16c804"}
Jul 09 17:31:05.630913 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:05.630785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjspn/must-gather-jrpcl" event={"ID":"06f90a3a-c2c7-4b42-a781-1559fde2ae76","Type":"ContainerStarted","Data":"0056debe48b1ec6a8d57bb5ebd7ea0a205404e16a758967ec09c0d2ce930c7e8"}
Jul 09 17:31:05.630913 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:05.630867    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjspn/must-gather-jrpcl" event={"ID":"06f90a3a-c2c7-4b42-a781-1559fde2ae76","Type":"ContainerStarted","Data":"4d3c68e167167a812090ec07527764e3e8165ae95ca8fcf832c5e753dc864f2d"}
Jul 09 17:31:05.646945 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:05.646863    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qjspn/must-gather-jrpcl" podStartSLOduration=0.971011158 podStartE2EDuration="1.646842718s" totalImagesPullingTime="675.83156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:31:05.646646268 +0000 UTC m=+2406.343818597" watchObservedRunningTime="2026-07-09 17:31:05.646842718 +0000 UTC m=+2406.344015048"
Jul 09 17:31:06.883011 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:06.882963    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4fx9l_77fbfceb-30f2-4e00-9a5f-c083f062c845/global-pull-secret-syncer/0.log"
Jul 09 17:31:07.071815 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:07.071761    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vp2jf_b895e80b-97a9-499b-bc6f-2d10ede4be8c/konnectivity-agent/0.log"
Jul 09 17:31:07.095916 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:07.095886    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-131.ec2.internal_bce01f930a010af7670ca85e9b0780aa/haproxy/0.log"
Jul 09 17:31:12.886587 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:12.886057    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f57fdb9f-hw4v8_060e6d2b-1b47-42b4-a547-953cacc3b41f/authorino/0.log"
Jul 09 17:31:13.024400 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:13.024361    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4s49b_0209346f-8206-4716-a6bf-d1631786923c/kuadrant-console-plugin/0.log"
Jul 09 17:31:13.257995 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:13.257897    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fcks8_6c9bed73-2881-4ba0-bc0f-d759bab1e883/limitador/0.log"
Jul 09 17:31:15.178126 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.178097    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fk57g_4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479/kube-state-metrics/0.log"
Jul 09 17:31:15.216980 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.216942    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fk57g_4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479/kube-rbac-proxy-main/0.log"
Jul 09 17:31:15.261141 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.261053    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fk57g_4a5c1fd2-e143-4c4d-9dbd-3e9ecdd08479/kube-rbac-proxy-self/0.log"
Jul 09 17:31:15.305369 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.305332    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8665cddf46-kq8wl_27e3c04e-029e-44be-bc68-cae624d998cd/metrics-server/0.log"
Jul 09 17:31:15.518865 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.518763    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94g6p_43f9e506-cc80-464f-9113-a8bbbffd1743/node-exporter/0.log"
Jul 09 17:31:15.551953 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.551919    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94g6p_43f9e506-cc80-464f-9113-a8bbbffd1743/kube-rbac-proxy/0.log"
Jul 09 17:31:15.576971 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.576937    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-94g6p_43f9e506-cc80-464f-9113-a8bbbffd1743/init-textfile/0.log"
Jul 09 17:31:15.674499 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.674413    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-gzpb4_d2683169-3272-4fda-9422-bb4e082a59da/kube-rbac-proxy-main/0.log"
Jul 09 17:31:15.696945 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.696915    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-gzpb4_d2683169-3272-4fda-9422-bb4e082a59da/kube-rbac-proxy-self/0.log"
Jul 09 17:31:15.717133 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.717096    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-gzpb4_d2683169-3272-4fda-9422-bb4e082a59da/openshift-state-metrics/0.log"
Jul 09 17:31:15.752249 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.752209    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/prometheus/0.log"
Jul 09 17:31:15.775762 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.775667    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/config-reloader/0.log"
Jul 09 17:31:15.806784 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.806745    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/thanos-sidecar/0.log"
Jul 09 17:31:15.828406 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.828379    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/kube-rbac-proxy-web/0.log"
Jul 09 17:31:15.848358 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.848331    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/kube-rbac-proxy/0.log"
Jul 09 17:31:15.869893 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.869867    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/kube-rbac-proxy-thanos/0.log"
Jul 09 17:31:15.892037 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:15.892013    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_187f2790-8a81-47f8-80a0-1cf05558c9e0/init-config-reloader/0.log"
Jul 09 17:31:16.058025 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.057930    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d56995d65-8sxml_87a7410d-336a-4e97-9507-f7432751a933/thanos-query/0.log"
Jul 09 17:31:16.077758 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.077726    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d56995d65-8sxml_87a7410d-336a-4e97-9507-f7432751a933/kube-rbac-proxy-web/0.log"
Jul 09 17:31:16.098535 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.098503    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d56995d65-8sxml_87a7410d-336a-4e97-9507-f7432751a933/kube-rbac-proxy/0.log"
Jul 09 17:31:16.120933 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.120907    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d56995d65-8sxml_87a7410d-336a-4e97-9507-f7432751a933/prom-label-proxy/0.log"
Jul 09 17:31:16.140430 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.140390    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d56995d65-8sxml_87a7410d-336a-4e97-9507-f7432751a933/kube-rbac-proxy-rules/0.log"
Jul 09 17:31:16.159245 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.159202    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d56995d65-8sxml_87a7410d-336a-4e97-9507-f7432751a933/kube-rbac-proxy-metrics/0.log"
Jul 09 17:31:16.780097 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.780062    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"]
Jul 09 17:31:16.785437 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.785408    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.793558 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.793519    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"]
Jul 09 17:31:16.879594 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.879556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclvd\" (UniqueName: \"kubernetes.io/projected/279c45dc-c53f-4727-9a8d-5d3516fd3b76-kube-api-access-sclvd\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.879594 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.879596    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-sys\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.879835 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.879634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-lib-modules\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.879835 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.879675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-proc\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.879835 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.879704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-podres\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.980393 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.980345    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclvd\" (UniqueName: \"kubernetes.io/projected/279c45dc-c53f-4727-9a8d-5d3516fd3b76-kube-api-access-sclvd\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.980590 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.980405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-sys\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.980590 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.980450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-lib-modules\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.980590 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.980523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-proc\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.980590 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.980574    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-podres\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.980848 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.980767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-podres\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.981177 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.981149    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-sys\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.981305 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.981273    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-lib-modules\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.981372 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.981315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/279c45dc-c53f-4727-9a8d-5d3516fd3b76-proc\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:16.990060 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:16.990032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclvd\" (UniqueName: \"kubernetes.io/projected/279c45dc-c53f-4727-9a8d-5d3516fd3b76-kube-api-access-sclvd\") pod \"perf-node-gather-daemonset-dsm9b\" (UID: \"279c45dc-c53f-4727-9a8d-5d3516fd3b76\") " pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:17.100864 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:17.100818    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:17.307138 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:17.306059    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"]
Jul 09 17:31:17.313556 ip-10-0-128-131 kubenswrapper[2567]: W0709 17:31:17.313521    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod279c45dc_c53f_4727_9a8d_5d3516fd3b76.slice/crio-a15f3de4636f101070cd7047af604d23a7c8dc2e8ba4a8d2eb87f983ce384f2c WatchSource:0}: Error finding container a15f3de4636f101070cd7047af604d23a7c8dc2e8ba4a8d2eb87f983ce384f2c: Status 404 returned error can't find the container with id a15f3de4636f101070cd7047af604d23a7c8dc2e8ba4a8d2eb87f983ce384f2c
Jul 09 17:31:17.715302 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:17.715205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b" event={"ID":"279c45dc-c53f-4727-9a8d-5d3516fd3b76","Type":"ContainerStarted","Data":"73955c330d2865e78566cc93bf4b5ad2099528a03bcf1f2c85129335fd459c32"}
Jul 09 17:31:17.715302 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:17.715281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b" event={"ID":"279c45dc-c53f-4727-9a8d-5d3516fd3b76","Type":"ContainerStarted","Data":"a15f3de4636f101070cd7047af604d23a7c8dc2e8ba4a8d2eb87f983ce384f2c"}
Jul 09 17:31:17.715598 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:17.715337    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:17.733727 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:17.733666    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b" podStartSLOduration=1.733645671 podStartE2EDuration="1.733645671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:31:17.731765983 +0000 UTC m=+2418.428938312" watchObservedRunningTime="2026-07-09 17:31:17.733645671 +0000 UTC m=+2418.430818001"
Jul 09 17:31:20.058525 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:20.058433    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ccp7t_156322e4-dcff-40e1-97d6-d11f5fc4c904/dns/0.log"
Jul 09 17:31:20.082513 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:20.082485    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ccp7t_156322e4-dcff-40e1-97d6-d11f5fc4c904/kube-rbac-proxy/0.log"
Jul 09 17:31:20.170775 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:20.170745    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqmmt_9c8b2164-2ab2-4a8f-9549-1ebf5a50800e/dns-node-resolver/0.log"
Jul 09 17:31:20.856710 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:20.856682    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2fhkn_f4d2921f-10ef-49f7-a552-6e511cb4435f/node-ca/0.log"
Jul 09 17:31:22.092780 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:22.092747    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cc96589f5-jcvjk_7108effc-c7bd-4c82-9a09-c3fa30ef8102/kube-auth-proxy/0.log"
Jul 09 17:31:22.924922 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:22.924888    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jgqv6_ce335e21-41ff-45cc-99ab-fa94472931be/serve-healthcheck-canary/0.log"
Jul 09 17:31:23.558984 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:23.558957    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ljgb9_7820052a-8a08-454a-9f39-c861fc961261/kube-rbac-proxy/0.log"
Jul 09 17:31:23.578002 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:23.577963    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ljgb9_7820052a-8a08-454a-9f39-c861fc961261/exporter/0.log"
Jul 09 17:31:23.598050 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:23.598021    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ljgb9_7820052a-8a08-454a-9f39-c861fc961261/extractor/0.log"
Jul 09 17:31:23.734037 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:23.734000    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qjspn/perf-node-gather-daemonset-dsm9b"
Jul 09 17:31:25.881499 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:25.881462    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bjvcf_3f1d1892-8663-4908-b974-f70fc205b18f/manager/0.log"
Jul 09 17:31:26.141894 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:26.141862    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mvspw_8d834298-f377-43c7-ade1-e5e12c818d78/manager/0.log"
Jul 09 17:31:26.485118 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:26.485083    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789dddf866-mnvbm_e71bbb6e-28e9-490c-b09b-0d5256f061bf/manager/0.log"
Jul 09 17:31:26.505169 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:26.505138    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789dddf866-mnvbm_e71bbb6e-28e9-490c-b09b-0d5256f061bf/copy-manifests/0.log"
Jul 09 17:31:27.790521 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:27.790494    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-db6444945-knhg9_0b1e7f49-1803-44d3-977b-5bcae4fe831b/manager/0.log"
Jul 09 17:31:27.824029 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:27.824000    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cc96589f5-jcvjk_7108effc-c7bd-4c82-9a09-c3fa30ef8102/kube-auth-proxy/0.log"
Jul 09 17:31:29.290597 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:29.290524    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cc96589f5-jcvjk_7108effc-c7bd-4c82-9a09-c3fa30ef8102/kube-auth-proxy/0.log"
Jul 09 17:31:34.078861 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.078755    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9fdgf_6bc945b8-7b27-42e2-b286-4020fdbd916f/kube-multus/0.log"
Jul 09 17:31:34.102367 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.102340    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/kube-multus-additional-cni-plugins/0.log"
Jul 09 17:31:34.127548 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.127517    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/egress-router-binary-copy/0.log"
Jul 09 17:31:34.149279 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.149254    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/cni-plugins/0.log"
Jul 09 17:31:34.170697 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.170671    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/bond-cni-plugin/0.log"
Jul 09 17:31:34.192479 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.192454    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/routeoverride-cni/0.log"
Jul 09 17:31:34.214477 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.214431    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/whereabouts-cni-bincopy/0.log"
Jul 09 17:31:34.234854 ip-10-0-128-131 kubenswrapper[2567]: I0709 17:31:34.234831    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-brnqb_60d6296c-10c8-498f-b402-cfbf51a4f1b3/whereabouts-cni/0.log"