Jul 27 20:35:04.991575 ip-10-0-142-163 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 20:35:06.426774 ip-10-0-142-163 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:35:06.426774 ip-10-0-142-163 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 20:35:06.426774 ip-10-0-142-163 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:35:06.426774 ip-10-0-142-163 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 20:35:06.426774 ip-10-0-142-163 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:35:06.428635 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.428532    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 20:35:06.434700 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434683    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:35:06.434700 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434700    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434704    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434707    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434711    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434714    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434718    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434721    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434724    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434727    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434730    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434738    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434741    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434744    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434747    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434750    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434752    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434755    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434757    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434760    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:35:06.434769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434763    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434765    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434768    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434771    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434775    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434779    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434783    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434786    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434788    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434791    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434794    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434797    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434799    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434802    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434805    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434807    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434810    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434812    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434815    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:35:06.435343 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434817    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434819    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434822    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434825    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434829    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434833    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434835    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434838    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434841    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434844    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434847    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434849    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434852    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434854    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434857    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434860    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434863    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434865    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434868    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:35:06.435843 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434871    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434874    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434876    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434879    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434882    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434884    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434888    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434891    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434893    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434896    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434898    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434901    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434903    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434918    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434922    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434926    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434929    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434932    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434934    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434937    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:35:06.436344 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434940    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434942    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434945    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434947    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434950    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434953    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434955    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434958    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434961    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434965    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434968    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434971    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434974    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434977    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434980    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434983    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434985    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434989    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434991    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434994    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:35:06.436869 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.434996    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435737    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435745    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435748    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435750    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435753    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435756    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435758    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435761    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435763    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435766    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435768    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435771    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435773    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435776    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435780    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435784    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435788    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435792    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:35:06.437354 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435795    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435799    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435802    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435805    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435808    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435811    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435814    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435817    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435819    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435822    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435824    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435828    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435830    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435834    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435838    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435841    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435844    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435847    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435850    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:35:06.437838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435853    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435855    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435858    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435860    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435863    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435866    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435868    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435871    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435873    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435876    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435878    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435881    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435883    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435888    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435891    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435894    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435896    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435899    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435901    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435904    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:35:06.438316 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435907    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435910    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435912    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435915    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435918    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435921    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435924    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435928    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435930    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435933    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435935    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435938    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435941    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435943    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435946    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435948    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435951    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435953    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435956    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:35:06.438831 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435958    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435961    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435963    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435966    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435968    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435971    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435974    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435977    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435980    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435982    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435985    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435987    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435990    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435992    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435995    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.435997    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436000    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436002    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436005    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436008    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:35:06.439294 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436010    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436013    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436015    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436381    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436386    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436389    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436392    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436394    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436397    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436400    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436402    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436405    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436408    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436410    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436412    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436415    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436418    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436420    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436423    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436426    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:35:06.439810 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436429    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436431    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436434    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436436    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436439    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436441    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436444    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436446    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436449    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436451    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436455    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436459    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436461    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436464    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436466    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436469    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436471    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436474    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436476    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:35:06.440291 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436478    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436481    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436483    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436486    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436489    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436491    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436493    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436496    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436498    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436501    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436503    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436507    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436510    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436512    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436515    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436518    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436520    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436523    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436525    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436528    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:35:06.440786 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436530    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436533    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436535    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436538    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436540    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436559    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436562    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436565    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436568    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436570    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436573    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436576    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436578    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436583    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436585    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436588    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436591    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436597    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436600    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436603    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:35:06.441318 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436605    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436608    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436610    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436615    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436617    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436620    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436622    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436625    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436628    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436632    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436635    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436637    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436639    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436642    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436646    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436650    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436653    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436656    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436659    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:35:06.441841 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436662    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436666    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436669    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.436672    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436748    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436755    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436762    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436766    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436770    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436773    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436779    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436783    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436786    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436790    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436793    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436796    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436801    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436804    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436807    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436810    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436813    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436816    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436820    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 27 20:35:06.442307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436823    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436827    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436830    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436834    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436837    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436841    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436844    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436847    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436850    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436854    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436857    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436860    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436863    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436867    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436870    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436873    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436876    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436879    2581 flags.go:64] FLAG: --enable-server="true"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436882    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436887    2581 flags.go:64] FLAG: --event-burst="100"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436891    2581 flags.go:64] FLAG: --event-qps="50"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436894    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436897    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436900    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 27 20:35:06.442881 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436903    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436907    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436910    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436913    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436916    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436919    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436922    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436925    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436928    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436930    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436933    2581 flags.go:64] FLAG: --feature-gates=""
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436937    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436940    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436943    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436946    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436949    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436952    2581 flags.go:64] FLAG: --help="false"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436955    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436961    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436964    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436967    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436970    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436973    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436976    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 20:35:06.443532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436979    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436982    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436984    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436988    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436991    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436994    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.436997    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437000    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437003    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437007    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437010    2581 flags.go:64] FLAG: --lock-file=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437012    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437015    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437018    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437024    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437027    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437030    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437033    2581 flags.go:64] FLAG: --logging-format="text"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437036    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437039    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437042    2581 flags.go:64] FLAG: --manifest-url=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437045    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437050    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437053    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437057    2581 flags.go:64] FLAG: --max-pods="110"
Jul 27 20:35:06.444153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437060    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437063    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437066    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437069    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437073    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437076    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437078    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437086    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437088    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437091    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437095    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437098    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437103    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437106    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437109    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437112    2581 flags.go:64] FLAG: --port="10250"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437116    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437119    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d6eaf9ed25dea762"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437123    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437125    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437128    2581 flags.go:64] FLAG: --register-node="true"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437131    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437135    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437138    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 27 20:35:06.444891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437141    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437144    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437147    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437150    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437153    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437156    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437159    2581 flags.go:64] FLAG: --runonce="false"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437161    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437164    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437167    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437170    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437173    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437176    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437180    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437182    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437185    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437188    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437191    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437194    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437197    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437200    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437202    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437207    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437210    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437214    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 20:35:06.445874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437218    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437221    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437224    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437227    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437230    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437233    2581 flags.go:64] FLAG: --v="2"
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437237    2581 flags.go:64] FLAG: --version="false"
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437240    2581 flags.go:64] FLAG: --vmodule=""
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437244    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.437247    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437361    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437364    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437368    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437370    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437373    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437375    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437378    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437381    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437383    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437386    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437389    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437391    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:35:06.446926 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437394    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437397    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437399    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437401    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437405    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437408    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437411    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437413    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437416    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437420    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437422    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437425    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437427    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437430    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437433    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437435    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437438    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437440    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437443    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437445    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:35:06.447769 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437448    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437451    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437454    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437456    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437458    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437461    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437464    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437467    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437469    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437472    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437475    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437479    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437481    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437484    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437487    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437489    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437493    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437496    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437498    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437501    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:35:06.448431 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437503    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437507    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437510    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437512    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437515    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437518    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437520    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437523    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437526    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437528    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437531    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437533    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437536    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437539    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437541    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437557    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437560    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437562    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437565    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437567    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:35:06.449042 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437570    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437573    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437575    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437578    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437583    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437587    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437590    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437593    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437597    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437599    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437603    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437606    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437610    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437613    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437616    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437619    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437621    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437624    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437626    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:35:06.449698 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437629    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437631    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437634    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437637    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437639    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437642    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437644    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.437647    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.438252    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.445149    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.445170    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.445195    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.445205    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445370    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:35:06.450499 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445378    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445384    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445390    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445394    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445399    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445403    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445407    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445417    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445421    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445426    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445430    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445436    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445443    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445448    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445453    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445457    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445461    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445466    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445471    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445475    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:35:06.450921 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445485    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445489    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445494    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445498    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445502    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445506    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445510    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445515    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445520    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445525    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445530    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445535    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445544    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445569    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445573    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445577    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445582    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445586    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445591    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445596    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:35:06.451566 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445600    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445605    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445611    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445617    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445626    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445631    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445636    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445641    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445645    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445650    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445660    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445664    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445669    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445673    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.445678    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446218    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446385    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446393    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446398    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:35:06.452096 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446403    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446408    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446413    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446419    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446423    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446428    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446432    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446437    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446441    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446446    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446450    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446454    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446459    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446463    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446468    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446472    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446477    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446481    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446485    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446490    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:35:06.452694 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446494    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446498    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446502    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446507    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446511    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446516    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446520    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446524    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446528    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446535    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446539    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446566    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446574    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446580    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446585    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446590    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446596    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:35:06.453189 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446600    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.446610    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446807    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446815    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446821    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446826    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446830    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446834    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446840    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446844    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446849    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446853    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446858    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446862    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446867    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:35:06.453691 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446871    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446877    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446883    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446888    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446893    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446897    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446901    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446912    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446916    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446921    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446925    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446930    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446934    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446939    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446943    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446947    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446951    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446955    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446959    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:35:06.454082 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446964    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446968    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446975    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446980    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446985    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446990    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446994    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.446998    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447002    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447007    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447011    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447015    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447019    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447024    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447029    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447033    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447037    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447042    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447046    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:35:06.454654 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447051    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447056    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447060    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447064    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447069    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447073    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447077    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447081    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447085    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447090    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447094    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447098    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447102    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447106    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447111    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447115    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447119    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447123    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447127    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447131    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:35:06.455301 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447136    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447140    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447145    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447149    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447153    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447157    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447162    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447168    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447172    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447176    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447181    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447185    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447190    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447194    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447199    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447203    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447207    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447211    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447215    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447219    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447224    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:35:06.456111 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447228    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447232    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447236    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447241    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447245    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447249    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:06.447255    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.447263    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.447430    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.451104    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.452523    2581 server.go:1019] "Starting client certificate rotation"
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.452642    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.452672    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:35:06.457028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.455499    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:35:06.457500 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.457320    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:35:06.465620 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.465604    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 27 20:35:06.469561 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.469534    2581 log.go:25] "Validated CRI v1 image API"
Jul 27 20:35:06.470415 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.470398    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 20:35:06.475525 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.475495    2581 fs.go:135] Filesystem UUIDs: map[659060cd-13f8-4f88-9a58-0855ca68df88:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9965e4f1-17c2-45aa-8fb6-e410062bc934:/dev/nvme0n1p4]
Jul 27 20:35:06.475631 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.475523    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 20:35:06.480184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.480163    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:35:06.481374 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.481260    2581 manager.go:216] Machine: {Timestamp:2026-07-27 20:35:06.479036864 +0000 UTC m=+0.359185560 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099901 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279640e1611ee84e9493496ee9b45f SystemUUID:ec279640-e161-1ee8-4e94-93496ee9b45f BootID:9af4bc02-a209-4af4-90b1-aa56327f03ab Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c3:be:03:a9:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c3:be:03:a9:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:03:1e:44:cb:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 20:35:06.481374 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.481371    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 20:35:06.481520 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.481487    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 20:35:06.484480 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.484454    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 20:35:06.484627 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.484483    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 20:35:06.485206 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.485196    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 20:35:06.485240 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.485208    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 20:35:06.485240 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.485217    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:35:06.485917 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.485907    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:35:06.487306 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.487296    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 20:35:06.489157 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489145    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:35:06.489271 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489261    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 20:35:06.489307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489290    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 20:35:06.489307 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489299    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 20:35:06.489365 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489312    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 20:35:06.489365 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489322    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 27 20:35:06.489365 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489330    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 20:35:06.489865 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489848    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:35:06.489917 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.489876    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:35:06.493995 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.493973    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 20:35:06.495990 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.495976    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 20:35:06.496037 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.496007    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 20:35:06.497938 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497925    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497943    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497950    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497955    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497964    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497970    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497980    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497989    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.497998    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.498003    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.498011    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 20:35:06.498012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.498016    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 20:35:06.498734 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.498723    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 20:35:06.498765 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.498736    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 20:35:06.500308 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.500280    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 20:35:06.500390 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.500358    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 20:35:06.500477 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.500453    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l8hkg"
Jul 27 20:35:06.500882 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.500865    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 20:35:06.502144 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.502130    2581 server.go:1268] "Started kubelet"
Jul 27 20:35:06.502213 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.502189    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 20:35:06.502273 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.502212    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 20:35:06.502314 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.502283    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 20:35:06.503231 ip-10-0-142-163 systemd[1]: Started Kubernetes Kubelet.
Jul 27 20:35:06.504253 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.504225    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 20:35:06.504414 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.504395    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 20:35:06.506045 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.506025    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l8hkg"
Jul 27 20:35:06.507931 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.507913    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 20:35:06.513114 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.513100    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 20:35:06.513175 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.513103    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 20:35:06.513693 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.513472    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:06.513898 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.513880    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 20:35:06.513965 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.513901    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 20:35:06.513965 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.513887    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 20:35:06.514049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514040    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 20:35:06.514104 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514049    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 20:35:06.514200 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514183    2581 factory.go:153] Registering CRI-O factory
Jul 27 20:35:06.514200 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514201    2581 factory.go:223] Registration of the crio container factory successfully
Jul 27 20:35:06.514347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514253    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 20:35:06.514347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514265    2581 factory.go:55] Registering systemd factory
Jul 27 20:35:06.514347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514271    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 27 20:35:06.514347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514293    2581 factory.go:103] Registering Raw factory
Jul 27 20:35:06.514347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514306    2581 manager.go:1196] Started watching for new ooms in manager
Jul 27 20:35:06.514731 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.514714    2581 manager.go:349] Starting recovery of all containers
Jul 27 20:35:06.516436 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.516416    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:35:06.520541 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.520519    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-163.ec2.internal\" not found" node="ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.526753 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.526686    2581 manager.go:354] Recovery completed
Jul 27 20:35:06.531910 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.531896    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.533721 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.533706    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.533778 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.533735    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.533778 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.533746    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.534260 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.534247    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 20:35:06.534325 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.534260    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 20:35:06.534325 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.534267    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.534325 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.534275    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:35:06.535003 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.534970    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.535003 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.535001    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.535093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.535014    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.535941 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.535930    2581 policy_none.go:49] "None policy: Start"
Jul 27 20:35:06.535975 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.535946    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 20:35:06.535975 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.535955    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 20:35:06.537101 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.537091    2581 policy_none.go:47] "Start"
Jul 27 20:35:06.571313 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571153    2581 manager.go:337] "Starting Device Plugin manager"
Jul 27 20:35:06.571424 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.571403    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 20:35:06.571424 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571415    2581 server.go:87] "Starting device plugin registration server"
Jul 27 20:35:06.571695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571683    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 20:35:06.571754 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571696    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 20:35:06.571831 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571813    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 20:35:06.571945 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571916    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 20:35:06.571945 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.571927    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 20:35:06.572460 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.572440    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 20:35:06.572531 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.572497    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:06.672847 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.672765    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.674494 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.674265    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.674494 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.674294    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.674494 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.674305    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.674494 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.674332    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.682703 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.682683    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.682794 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.682709    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-163.ec2.internal\": node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:06.698746 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.698721    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:06.726498 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.726461    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 20:35:06.727676 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.727657    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 20:35:06.727758 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.727706    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 20:35:06.727758 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.727736    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 20:35:06.727849 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.727781    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 20:35:06.729943 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.729923    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:35:06.798829 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.798795    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:06.828472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.828448    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"]
Jul 27 20:35:06.828572 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.828523    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.829533 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.829518    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.829630 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.829565    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.829630 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.829590    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.831086 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831072    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.831213 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.831257 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831233    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.831875 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831860    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.831875 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831868    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.831982 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831886    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.831982 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831893    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.831982 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831898    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.831982 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.831903    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.833213 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.833200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.833257 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.833224    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:35:06.833927 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.833914    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:35:06.833994 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.833940    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:35:06.833994 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.833952    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:35:06.856360 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.856334    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-163.ec2.internal\" not found" node="ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.860779 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.860762    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-163.ec2.internal\" not found" node="ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.899812 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:06.899788    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:06.915673 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.915648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c738e99dd5f2475b57b372ca1b7098-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal\" (UID: \"87c738e99dd5f2475b57b372ca1b7098\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.915750 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.915676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c17c641c6802d8060ff85f0f666ff861-config\") pod \"kube-apiserver-proxy-ip-10-0-142-163.ec2.internal\" (UID: \"c17c641c6802d8060ff85f0f666ff861\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:06.915750 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:06.915694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c738e99dd5f2475b57b372ca1b7098-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal\" (UID: \"87c738e99dd5f2475b57b372ca1b7098\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.000297 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.000235    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.016610 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.016588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c738e99dd5f2475b57b372ca1b7098-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal\" (UID: \"87c738e99dd5f2475b57b372ca1b7098\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.016693 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.016616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c738e99dd5f2475b57b372ca1b7098-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal\" (UID: \"87c738e99dd5f2475b57b372ca1b7098\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.016693 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.016633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c17c641c6802d8060ff85f0f666ff861-config\") pod \"kube-apiserver-proxy-ip-10-0-142-163.ec2.internal\" (UID: \"c17c641c6802d8060ff85f0f666ff861\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.016693 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.016660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c17c641c6802d8060ff85f0f666ff861-config\") pod \"kube-apiserver-proxy-ip-10-0-142-163.ec2.internal\" (UID: \"c17c641c6802d8060ff85f0f666ff861\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.016693 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.016679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c738e99dd5f2475b57b372ca1b7098-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal\" (UID: \"87c738e99dd5f2475b57b372ca1b7098\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.016828 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.016687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87c738e99dd5f2475b57b372ca1b7098-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal\" (UID: \"87c738e99dd5f2475b57b372ca1b7098\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.100744 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.100715    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.159285 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.159252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.164388 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.164370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:07.201078 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.201035    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.301649 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.301563    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.402127 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.402099    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.451634 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.451602    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 20:35:07.452138 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.451762    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:35:07.452138 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.451805    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:35:07.502764 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.502731    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.511266 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.511235    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:30:06 +0000 UTC" deadline="2028-02-22 19:44:33.825481165 +0000 UTC"
Jul 27 20:35:07.511266 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.511263    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13799h9m26.314220873s"
Jul 27 20:35:07.603195 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.603167    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.667808 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.667781    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:35:07.703907 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.703868    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.732109 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:07.732081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17c641c6802d8060ff85f0f666ff861.slice/crio-ab6260860e59a8fa6e85617cb877863bcdfc73c13d09fb738158df28316381fd WatchSource:0}: Error finding container ab6260860e59a8fa6e85617cb877863bcdfc73c13d09fb738158df28316381fd: Status 404 returned error can't find the container with id ab6260860e59a8fa6e85617cb877863bcdfc73c13d09fb738158df28316381fd
Jul 27 20:35:07.736855 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.736839    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:35:07.804742 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.804711    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.905318 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:07.905254    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-163.ec2.internal\" not found"
Jul 27 20:35:07.951586 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:07.951564    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:35:07.970372 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:07.970347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c738e99dd5f2475b57b372ca1b7098.slice/crio-91fd4ffbe3dcb8f298cc32aec2925892b6ba12fd19b6d12f9404ac2a56286244 WatchSource:0}: Error finding container 91fd4ffbe3dcb8f298cc32aec2925892b6ba12fd19b6d12f9404ac2a56286244: Status 404 returned error can't find the container with id 91fd4ffbe3dcb8f298cc32aec2925892b6ba12fd19b6d12f9404ac2a56286244
Jul 27 20:35:08.014608 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.014582    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:08.024449 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.024425    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:35:08.025731 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.025715    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal"
Jul 27 20:35:08.032986 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.032966    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:35:08.458076 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.458044    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:35:08.491032 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.490979    2581 apiserver.go:52] "Watching apiserver"
Jul 27 20:35:08.499232 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.499203    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 20:35:08.500407 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.500380    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-h8rgc","openshift-dns/node-resolver-wz462","openshift-image-registry/node-ca-2jbj4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal","openshift-multus/multus-additional-cni-plugins-hjlgd","openshift-network-diagnostics/network-check-target-htcjh","openshift-network-operator/iptables-alerter-bgzwj","openshift-ovn-kubernetes/ovnkube-node-fq5fk","kube-system/global-pull-secret-syncer-p4zdq","kube-system/konnectivity-agent-zj2jj","kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw","openshift-multus/multus-6mrzj","openshift-multus/network-metrics-daemon-7qvwm"]
Jul 27 20:35:08.502146 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.502025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.503497 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.503476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.504114 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.504093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5jnc\""
Jul 27 20:35:08.504213 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.504170    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 20:35:08.504213 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.504189    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 20:35:08.504935 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.504632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.506530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.506050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 20:35:08.506530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.506142    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.506530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.506160    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 20:35:08.506530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.506359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.506530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.506405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z2vk7\""
Jul 27 20:35:08.506895 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.506597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:08.506895 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.506663    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:08.507528 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.507078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.507528 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.507211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mx4h5\""
Jul 27 20:35:08.507528 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.507272    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 20:35:08.507528 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.507517    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.507820 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.507609    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.508417 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.508399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:08.508504 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.508464    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:08.509028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.509007    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 20:35:08.509770 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.509372    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sjvtx\""
Jul 27 20:35:08.509770 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.509613    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 20:35:08.509770 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.509623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.509770 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.509680    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:08.512630 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.512532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.514356 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.513482    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 20:35:08.514356 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.514065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.514636 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.514617    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.514714 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.514670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 20:35:08.514880 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.514861    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.515003 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.514984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8jmkm\""
Jul 27 20:35:08.515439 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.515420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.516301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.516111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5pq2v\""
Jul 27 20:35:08.516301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.516149    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.516301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.516157    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.517251 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.516859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.518149 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.518129    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.518241 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.518229    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.518398 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.518378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-465t8\""
Jul 27 20:35:08.518494 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.518379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.519304 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.519183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 20:35:08.519304 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.519281    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8bx6\""
Jul 27 20:35:08.521089 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.519941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 20:35:08.521611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.521591    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.523804 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.522346    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 20:35:08.523804 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.523254    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 20:35:08.523804 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.523268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.523804 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.523622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tnlvr\""
Jul 27 20:35:08.524015 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.523851    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 20:35:08.524061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 20:35:08.524237 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-netns\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524304 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/421f2241-7f4c-446b-ae1a-0fd24b7046d7-kubelet-config\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.524304 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524277    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 20:35:08.524304 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.524437 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-cnibin\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524437 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-hostroot\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524533 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61ac49fe-a4a8-4fce-b478-badc8674f5f0-cni-binary-copy\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524533 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-daemon-config\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524643 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-multus-certs\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524692 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsj9l\" (UniqueName: \"kubernetes.io/projected/61ac49fe-a4a8-4fce-b478-badc8674f5f0-kube-api-access-qsj9l\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524743 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-run\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.524794 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-host\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.524844 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-cnibin\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.524893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-k8s-cni-cncf-io\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.524943 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:08.524943 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524903    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:35:08.524943 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7121382-96d9-460f-bfd9-bdfa44a6b64b-tmp-dir\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.525135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.524981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-cni-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.525135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-cni-multus\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.525135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.525135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rmz\" (UniqueName: \"kubernetes.io/projected/f7121382-96d9-460f-bfd9-bdfa44a6b64b-kube-api-access-k7rmz\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.525135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-kubelet\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.525135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-host\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.525430 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysconfig\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.525430 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-lib-modules\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.525430 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/120627ab-6ae5-48fc-9989-6de22a822e10-etc-tuned\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.525743 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-cni-bin\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.525807 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-systemd\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.525807 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsbr\" (UniqueName: \"kubernetes.io/projected/f35da8cb-24bc-4c87-9821-df562697ed0c-kube-api-access-qdsbr\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:08.525896 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjw7q\" (UniqueName: \"kubernetes.io/projected/1d9bde2c-f800-4c98-8921-61e3847d82d8-kube-api-access-cjw7q\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.525896 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-kubernetes\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.525896 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-system-cni-dir\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.526031 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d9bde2c-f800-4c98-8921-61e3847d82d8-iptables-alerter-script\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.526031 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2-konnectivity-ca\") pod \"konnectivity-agent-zj2jj\" (UID: \"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2\") " pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.526031 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.525972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5cv\" (UniqueName: \"kubernetes.io/projected/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-kube-api-access-nr5cv\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.526175 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.526222 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p6kz\" (UniqueName: \"kubernetes.io/projected/b03067a3-a707-417d-b3b5-eeeea933c906-kube-api-access-5p6kz\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.526222 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-system-cni-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.526315 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysctl-d\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.526315 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-os-release\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.526315 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.526456 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:08.526456 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-var-lib-kubelet\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.526456 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-conf-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.526623 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-etc-kubernetes\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.526623 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl9fr\" (UniqueName: \"kubernetes.io/projected/120627ab-6ae5-48fc-9989-6de22a822e10-kube-api-access-fl9fr\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.526736 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.526578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-os-release\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d9bde2c-f800-4c98-8921-61e3847d82d8-host-slash\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7121382-96d9-460f-bfd9-bdfa44a6b64b-hosts-file\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-serviceca\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysctl-conf\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-sys\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/120627ab-6ae5-48fc-9989-6de22a822e10-tmp\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2-agent-certs\") pod \"konnectivity-agent-zj2jj\" (UID: \"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2\") " pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-socket-dir-parent\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.527378 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-modprobe-d\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.527924 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.527403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/421f2241-7f4c-446b-ae1a-0fd24b7046d7-dbus\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.557919 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.557894    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4jhcw"
Jul 27 20:35:08.565142 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.565117    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4jhcw"
Jul 27 20:35:08.614854 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.614783    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 20:35:08.627682 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-run\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.627825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-host\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.627825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-cnibin\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-run\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-host\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-k8s-cni-cncf-io\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7121382-96d9-460f-bfd9-bdfa44a6b64b-tmp-dir\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-cni-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-cnibin\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.627973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-cni-multus\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-slash\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-systemd\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-var-lib-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.628057    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-etc-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628111 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.628130    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:35:09.128096687 +0000 UTC m=+3.008245384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rmz\" (UniqueName: \"kubernetes.io/projected/f7121382-96d9-460f-bfd9-bdfa44a6b64b-kube-api-access-k7rmz\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-kubelet\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-host\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-systemd-units\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-node-log\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-cni-netd\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysconfig\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-lib-modules\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/120627ab-6ae5-48fc-9989-6de22a822e10-etc-tuned\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-cni-bin\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-env-overrides\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmlb9\" (UniqueName: \"kubernetes.io/projected/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-kube-api-access-vmlb9\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-systemd\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.628626 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsbr\" (UniqueName: \"kubernetes.io/projected/f35da8cb-24bc-4c87-9821-df562697ed0c-kube-api-access-qdsbr\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjw7q\" (UniqueName: \"kubernetes.io/projected/1d9bde2c-f800-4c98-8921-61e3847d82d8-kube-api-access-cjw7q\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-kubernetes\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-system-cni-dir\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d9bde2c-f800-4c98-8921-61e3847d82d8-iptables-alerter-script\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2-konnectivity-ca\") pod \"konnectivity-agent-zj2jj\" (UID: \"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2\") " pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5cv\" (UniqueName: \"kubernetes.io/projected/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-kube-api-access-nr5cv\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovnkube-script-lib\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p6kz\" (UniqueName: \"kubernetes.io/projected/b03067a3-a707-417d-b3b5-eeeea933c906-kube-api-access-5p6kz\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-system-cni-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-ovn\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.628976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-socket-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysctl-d\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-os-release\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.629312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-var-lib-kubelet\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d9bde2c-f800-4c98-8921-61e3847d82d8-iptables-alerter-script\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-conf-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-k8s-cni-cncf-io\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-conf-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7121382-96d9-460f-bfd9-bdfa44a6b64b-tmp-dir\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.629601    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.629653    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:09.129637958 +0000 UTC m=+3.009786663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysctl-d\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-os-release\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-var-lib-kubelet\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-system-cni-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-cni-bin\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.629972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-cni-dir\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-kubelet\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-var-lib-cni-multus\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-host\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysconfig\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-lib-modules\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-system-cni-dir\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-kubernetes\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-etc-kubernetes\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630520    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl9fr\" (UniqueName: \"kubernetes.io/projected/120627ab-6ae5-48fc-9989-6de22a822e10-kube-api-access-fl9fr\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-os-release\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d9bde2c-f800-4c98-8921-61e3847d82d8-host-slash\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7121382-96d9-460f-bfd9-bdfa44a6b64b-hosts-file\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-serviceca\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-cni-bin\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-systemd\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.630802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovn-node-metrics-cert\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-os-release\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2-konnectivity-ca\") pod \"konnectivity-agent-zj2jj\" (UID: \"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2\") " pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d9bde2c-f800-4c98-8921-61e3847d82d8-host-slash\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7121382-96d9-460f-bfd9-bdfa44a6b64b-hosts-file\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.630978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-registration-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysctl-conf\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-sys\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/120627ab-6ae5-48fc-9989-6de22a822e10-tmp\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2-agent-certs\") pod \"konnectivity-agent-zj2jj\" (UID: \"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2\") " pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-socket-dir-parent\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovnkube-config\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-modprobe-d\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-kubelet\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-log-socket\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-serviceca\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.631850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-etc-selinux\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/421f2241-7f4c-446b-ae1a-0fd24b7046d7-dbus\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-device-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-netns\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/421f2241-7f4c-446b-ae1a-0fd24b7046d7-kubelet-config\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-run-netns\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgnl\" (UniqueName: \"kubernetes.io/projected/0b33dce1-e944-4d01-b8df-854f98434262-kube-api-access-gfgnl\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-socket-dir-parent\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-cnibin\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-hostroot\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-sys-fs\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61ac49fe-a4a8-4fce-b478-badc8674f5f0-cni-binary-copy\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-daemon-config\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-multus-certs\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-netns\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.632668 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsj9l\" (UniqueName: \"kubernetes.io/projected/61ac49fe-a4a8-4fce-b478-badc8674f5f0-kube-api-access-qsj9l\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b03067a3-a707-417d-b3b5-eeeea933c906-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-modprobe-d\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/421f2241-7f4c-446b-ae1a-0fd24b7046d7-dbus\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-etc-sysctl-conf\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/120627ab-6ae5-48fc-9989-6de22a822e10-sys\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.631997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-etc-kubernetes\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-cnibin\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/421f2241-7f4c-446b-ae1a-0fd24b7046d7-kubelet-config\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-host-run-multus-certs\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61ac49fe-a4a8-4fce-b478-badc8674f5f0-hostroot\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b03067a3-a707-417d-b3b5-eeeea933c906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632621    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61ac49fe-a4a8-4fce-b478-badc8674f5f0-multus-daemon-config\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.633459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.632627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61ac49fe-a4a8-4fce-b478-badc8674f5f0-cni-binary-copy\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.635424 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.635266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/120627ab-6ae5-48fc-9989-6de22a822e10-etc-tuned\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.635607 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.635579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2-agent-certs\") pod \"konnectivity-agent-zj2jj\" (UID: \"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2\") " pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.635854 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.635835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/120627ab-6ae5-48fc-9989-6de22a822e10-tmp\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.639502 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.639481    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:08.639601 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.639511    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:08.639601 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.639525    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:08.639601 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:08.639600    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:09.139583364 +0000 UTC m=+3.019732064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:08.645327 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.644978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p6kz\" (UniqueName: \"kubernetes.io/projected/b03067a3-a707-417d-b3b5-eeeea933c906-kube-api-access-5p6kz\") pod \"multus-additional-cni-plugins-hjlgd\" (UID: \"b03067a3-a707-417d-b3b5-eeeea933c906\") " pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.645327 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.645164    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:35:08.646732 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.646711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsj9l\" (UniqueName: \"kubernetes.io/projected/61ac49fe-a4a8-4fce-b478-badc8674f5f0-kube-api-access-qsj9l\") pod \"multus-6mrzj\" (UID: \"61ac49fe-a4a8-4fce-b478-badc8674f5f0\") " pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.646944 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.646867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjw7q\" (UniqueName: \"kubernetes.io/projected/1d9bde2c-f800-4c98-8921-61e3847d82d8-kube-api-access-cjw7q\") pod \"iptables-alerter-bgzwj\" (UID: \"1d9bde2c-f800-4c98-8921-61e3847d82d8\") " pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.647165 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.647124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl9fr\" (UniqueName: \"kubernetes.io/projected/120627ab-6ae5-48fc-9989-6de22a822e10-kube-api-access-fl9fr\") pod \"tuned-h8rgc\" (UID: \"120627ab-6ae5-48fc-9989-6de22a822e10\") " pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.647165 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.647136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rmz\" (UniqueName: \"kubernetes.io/projected/f7121382-96d9-460f-bfd9-bdfa44a6b64b-kube-api-access-k7rmz\") pod \"node-resolver-wz462\" (UID: \"f7121382-96d9-460f-bfd9-bdfa44a6b64b\") " pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.647366 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.647348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsbr\" (UniqueName: \"kubernetes.io/projected/f35da8cb-24bc-4c87-9821-df562697ed0c-kube-api-access-qdsbr\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:08.651077 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.651039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5cv\" (UniqueName: \"kubernetes.io/projected/8a8bba74-5af3-4e93-bfdd-b8d35c760fa7-kube-api-access-nr5cv\") pod \"node-ca-2jbj4\" (UID: \"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7\") " pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.732199 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732199 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.732199 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-cni-bin\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovn-node-metrics-cert\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-registration-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovnkube-config\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-kubelet\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-log-socket\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-etc-selinux\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-device-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-run-netns\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgnl\" (UniqueName: \"kubernetes.io/projected/0b33dce1-e944-4d01-b8df-854f98434262-kube-api-access-gfgnl\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.732472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-sys-fs\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-slash\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-systemd\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-var-lib-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-etc-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-systemd-units\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-node-log\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-cni-netd\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-env-overrides\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmlb9\" (UniqueName: \"kubernetes.io/projected/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-kube-api-access-vmlb9\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-slash\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovnkube-script-lib\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-sys-fs\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-ovn\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-socket-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733061 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.732972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-socket-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-cni-bin\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-kubelet\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-log-socket\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-node-log\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-etc-selinux\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-registration-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b33dce1-e944-4d01-b8df-854f98434262-device-dir\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-ovn\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-var-lib-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovnkube-script-lib\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-run-systemd\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-etc-openvswitch\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-run-netns\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-host-cni-netd\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.733939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.733542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-systemd-units\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.734736 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.734008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovnkube-config\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.734736 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.734266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-env-overrides\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.734941 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.734883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal" event={"ID":"87c738e99dd5f2475b57b372ca1b7098","Type":"ContainerStarted","Data":"91fd4ffbe3dcb8f298cc32aec2925892b6ba12fd19b6d12f9404ac2a56286244"}
Jul 27 20:35:08.736279 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.736257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-ovn-node-metrics-cert\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.736392 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.736271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal" event={"ID":"c17c641c6802d8060ff85f0f666ff861","Type":"ContainerStarted","Data":"ab6260860e59a8fa6e85617cb877863bcdfc73c13d09fb738158df28316381fd"}
Jul 27 20:35:08.741845 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.741823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmlb9\" (UniqueName: \"kubernetes.io/projected/f31ad677-ac5d-4f18-b038-3f6505a6dfa2-kube-api-access-vmlb9\") pod \"ovnkube-node-fq5fk\" (UID: \"f31ad677-ac5d-4f18-b038-3f6505a6dfa2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.742096 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.742070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgnl\" (UniqueName: \"kubernetes.io/projected/0b33dce1-e944-4d01-b8df-854f98434262-kube-api-access-gfgnl\") pod \"aws-ebs-csi-driver-node-wttdw\" (UID: \"0b33dce1-e944-4d01-b8df-854f98434262\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:08.815854 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.815822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:08.828328 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.828306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6mrzj"
Jul 27 20:35:08.837194 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.837172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2jbj4"
Jul 27 20:35:08.842028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.842007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hjlgd"
Jul 27 20:35:08.850627 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.850609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bgzwj"
Jul 27 20:35:08.859180 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.859154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wz462"
Jul 27 20:35:08.865664 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.865640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8rgc"
Jul 27 20:35:08.874396 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.874375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:08.880158 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:08.880138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw"
Jul 27 20:35:09.136146 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.136112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:09.136324 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.136180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:09.136324 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.136271    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:09.136324 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.136278    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:09.136477 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.136340    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:35:10.136320867 +0000 UTC m=+4.016469563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:09.136477 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.136361    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:10.136351357 +0000 UTC m=+4.016500062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:09.236682 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.236649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:09.236859 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.236830    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:09.236859 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.236854    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:09.236962 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.236869    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:09.236962 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:09.236929    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:10.236910623 +0000 UTC m=+4.117059305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:09.567625 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.567519    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:30:08 +0000 UTC" deadline="2028-03-03 15:38:13.240076757 +0000 UTC"
Jul 27 20:35:09.567625 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.567574    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14035h3m3.672506029s"
Jul 27 20:35:09.648415 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:09.648387    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ac49fe_a4a8_4fce_b478_badc8674f5f0.slice/crio-c1394ec44e4231526ca3e491deaf44a3d75b88223fdcb22a1147617ee4993cf5 WatchSource:0}: Error finding container c1394ec44e4231526ca3e491deaf44a3d75b88223fdcb22a1147617ee4993cf5: Status 404 returned error can't find the container with id c1394ec44e4231526ca3e491deaf44a3d75b88223fdcb22a1147617ee4993cf5
Jul 27 20:35:09.651414 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:09.651392    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03067a3_a707_417d_b3b5_eeeea933c906.slice/crio-0121e63053d96cb0c009dabae141ef7c591f61315a97c4556c41e9b5b9277356 WatchSource:0}: Error finding container 0121e63053d96cb0c009dabae141ef7c591f61315a97c4556c41e9b5b9277356: Status 404 returned error can't find the container with id 0121e63053d96cb0c009dabae141ef7c591f61315a97c4556c41e9b5b9277356
Jul 27 20:35:09.652987 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:09.652820    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdf31f2_f174_4794_8bd5_2ee8bfa7d8c2.slice/crio-12f3cb1bcf061e089bb2b97d5c73828177c10409ffd1f8454f4cd65eb80f9299 WatchSource:0}: Error finding container 12f3cb1bcf061e089bb2b97d5c73828177c10409ffd1f8454f4cd65eb80f9299: Status 404 returned error can't find the container with id 12f3cb1bcf061e089bb2b97d5c73828177c10409ffd1f8454f4cd65eb80f9299
Jul 27 20:35:09.653893 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:09.653739    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120627ab_6ae5_48fc_9989_6de22a822e10.slice/crio-e9190c73efb07b9e20ff3c5ff914700174c5f0eab3afa4fa608a33e86a368879 WatchSource:0}: Error finding container e9190c73efb07b9e20ff3c5ff914700174c5f0eab3afa4fa608a33e86a368879: Status 404 returned error can't find the container with id e9190c73efb07b9e20ff3c5ff914700174c5f0eab3afa4fa608a33e86a368879
Jul 27 20:35:09.654782 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:09.654744    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b33dce1_e944_4d01_b8df_854f98434262.slice/crio-eb9a1801e760216686c271a9f5b2948694eed8981bf9926e1dfdc6d8ed5eb762 WatchSource:0}: Error finding container eb9a1801e760216686c271a9f5b2948694eed8981bf9926e1dfdc6d8ed5eb762: Status 404 returned error can't find the container with id eb9a1801e760216686c271a9f5b2948694eed8981bf9926e1dfdc6d8ed5eb762
Jul 27 20:35:09.659726 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:09.659454    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31ad677_ac5d_4f18_b038_3f6505a6dfa2.slice/crio-f0b5b3097e8afc02b5995583d607bed951c68d076e2ddde80825f5d8f306771e WatchSource:0}: Error finding container f0b5b3097e8afc02b5995583d607bed951c68d076e2ddde80825f5d8f306771e: Status 404 returned error can't find the container with id f0b5b3097e8afc02b5995583d607bed951c68d076e2ddde80825f5d8f306771e
Jul 27 20:35:09.739335 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.739166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"f0b5b3097e8afc02b5995583d607bed951c68d076e2ddde80825f5d8f306771e"}
Jul 27 20:35:09.740086 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.740065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wz462" event={"ID":"f7121382-96d9-460f-bfd9-bdfa44a6b64b","Type":"ContainerStarted","Data":"68c42a4aa7a406c7bd95e8c1e753f8467c2cbfa12f92e77144f6ebc82b7de1e0"}
Jul 27 20:35:09.742819 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.742796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw" event={"ID":"0b33dce1-e944-4d01-b8df-854f98434262","Type":"ContainerStarted","Data":"eb9a1801e760216686c271a9f5b2948694eed8981bf9926e1dfdc6d8ed5eb762"}
Jul 27 20:35:09.747244 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.747222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8rgc" event={"ID":"120627ab-6ae5-48fc-9989-6de22a822e10","Type":"ContainerStarted","Data":"e9190c73efb07b9e20ff3c5ff914700174c5f0eab3afa4fa608a33e86a368879"}
Jul 27 20:35:09.750874 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.750849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zj2jj" event={"ID":"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2","Type":"ContainerStarted","Data":"12f3cb1bcf061e089bb2b97d5c73828177c10409ffd1f8454f4cd65eb80f9299"}
Jul 27 20:35:09.752372 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.752351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bgzwj" event={"ID":"1d9bde2c-f800-4c98-8921-61e3847d82d8","Type":"ContainerStarted","Data":"123546f22a072ccb29e474f416ba7fe7583a35c843a2d1ec5d86a2570854c618"}
Jul 27 20:35:09.755870 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.755841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal" event={"ID":"c17c641c6802d8060ff85f0f666ff861","Type":"ContainerStarted","Data":"30dde273fbda39b80cfb411bb51313ffe9fc4a82f4d1e39b6b21cb31e4c750c4"}
Jul 27 20:35:09.756898 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.756875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2jbj4" event={"ID":"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7","Type":"ContainerStarted","Data":"3fe6c3c3f745bfd3e2c320665964c69b372622f121a67013dd83dc2159c47b67"}
Jul 27 20:35:09.757845 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.757827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerStarted","Data":"0121e63053d96cb0c009dabae141ef7c591f61315a97c4556c41e9b5b9277356"}
Jul 27 20:35:09.759212 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.759190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6mrzj" event={"ID":"61ac49fe-a4a8-4fce-b478-badc8674f5f0","Type":"ContainerStarted","Data":"c1394ec44e4231526ca3e491deaf44a3d75b88223fdcb22a1147617ee4993cf5"}
Jul 27 20:35:09.769816 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:09.769769    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-163.ec2.internal" podStartSLOduration=1.769757449 podStartE2EDuration="1.769757449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:09.769676952 +0000 UTC m=+3.649825657" watchObservedRunningTime="2026-07-27 20:35:09.769757449 +0000 UTC m=+3.649906154"
Jul 27 20:35:10.142877 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.142843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:10.143046 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.142917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:10.143120 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.143060    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:10.143182 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.143121    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:12.143102583 +0000 UTC m=+6.023251270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:10.143500 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.143481    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:10.143593 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.143536    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:35:12.143520154 +0000 UTC m=+6.023668844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:10.244223 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.244191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:10.244382 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.244352    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:10.244382 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.244369    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:10.244382 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.244381    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:10.244603 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.244435    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:12.244416991 +0000 UTC m=+6.124565692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:10.567890 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.567814    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:30:08 +0000 UTC" deadline="2028-02-10 00:59:54.849862284 +0000 UTC"
Jul 27 20:35:10.567890 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.567889    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13492h24m44.281978314s"
Jul 27 20:35:10.731807 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.730953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:10.731807 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.731077    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:10.731807 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.731493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:10.731807 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.731598    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:10.731807 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.731662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:10.731807 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:10.731744    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:10.776237 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.775021    2581 generic.go:356] "Generic (PLEG): container finished" podID="87c738e99dd5f2475b57b372ca1b7098" containerID="42541e522779d1262ed9e2b2ee186f7474ca1917e3247cbb85fa1945f19b3a25" exitCode=0
Jul 27 20:35:10.776237 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:10.775994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal" event={"ID":"87c738e99dd5f2475b57b372ca1b7098","Type":"ContainerDied","Data":"42541e522779d1262ed9e2b2ee186f7474ca1917e3247cbb85fa1945f19b3a25"}
Jul 27 20:35:11.784836 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:11.784800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal" event={"ID":"87c738e99dd5f2475b57b372ca1b7098","Type":"ContainerStarted","Data":"018dc7a68ed304640fd04c5bb4998c8051ecb418e3981a3a6e53c43874eefea1"}
Jul 27 20:35:12.161831 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:12.161770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:12.162033 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:12.161839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:12.162033 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.161981    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:12.162159 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.162039    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:16.162021427 +0000 UTC m=+10.042170111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:12.162497 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.162449    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:12.162646 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.162505    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:35:16.162488632 +0000 UTC m=+10.042637319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:12.263032 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:12.262933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:12.263217 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.263117    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:12.263217 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.263138    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:12.263217 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.263152    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:12.263217 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.263210    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:16.263193656 +0000 UTC m=+10.143342339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:12.731323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:12.731224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:12.731516 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.731356    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:12.731762 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:12.731741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:12.731859 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.731839    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:12.731920 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:12.731901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:12.732000 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:12.731985    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:14.728483 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:14.728402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:14.728935 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:14.728412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:14.728935 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:14.728561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:14.728935 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:14.728541    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:14.728935 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:14.728652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:14.728935 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:14.728728    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:16.195341 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:16.195141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:16.195341 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:16.195210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:16.195341 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.195316    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:16.195341 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.195332    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:16.195996 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.195399    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:35:24.195373843 +0000 UTC m=+18.075522548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:16.195996 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.195422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:24.195412213 +0000 UTC m=+18.075560912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:16.296479 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:16.295802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:16.296479 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.296015    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:16.296479 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.296033    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:16.296479 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.296048    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:16.296479 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.296105    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:24.296085583 +0000 UTC m=+18.176234267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:16.731034 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:16.730101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:16.731034 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.730273    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:16.731034 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:16.730831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:16.731034 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.730992    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:16.731034 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:16.731044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:16.731490 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:16.731138    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:18.731430 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:18.731385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:18.731928 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:18.731521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:18.731928 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:18.731527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:18.731928 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:18.731529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:18.731928 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:18.731666    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:18.731928 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:18.731755    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:20.730800 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:20.730682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:20.730800 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:20.730803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:20.731305 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:20.730815    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:20.731305 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:20.730857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:20.731305 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:20.730989    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:20.731305 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:20.731076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:22.728460 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:22.728423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:22.728928 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:22.728470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:22.728928 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:22.728430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:22.728928 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:22.728580    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:22.728928 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:22.728671    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:22.728928 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:22.728739    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:24.255855 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:24.255821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:24.256395 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:24.255884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:24.256395 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.256035    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:24.256395 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.256098    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:40.256080282 +0000 UTC m=+34.136228965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:24.256616 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.256461    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:24.256616 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.256530    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:35:40.256511276 +0000 UTC m=+34.136659982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:24.357008 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:24.356965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:24.357176 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.357135    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:24.357176 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.357160    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:24.357176 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.357175    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:24.357307 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.357239    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:40.357220546 +0000 UTC m=+34.237369252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:24.728947 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:24.728912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:24.729130 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:24.728912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:24.729130 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.729048    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:24.729130 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:24.728915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:24.729130 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.729098    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:24.729345 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:24.729206    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:26.729913 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:26.729882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:26.730345 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:26.729971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:26.730345 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:26.729999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:26.730345 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:26.730017    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:26.730345 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:26.730082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:26.730345 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:26.730141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:28.729005 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.728977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:28.729701 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.729024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:28.729701 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.728977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:28.729701 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:28.729112    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:28.729701 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:28.729199    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:28.729701 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:28.729276    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:28.825409 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.825245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6mrzj" event={"ID":"61ac49fe-a4a8-4fce-b478-badc8674f5f0","Type":"ContainerStarted","Data":"85e07b5f73ff145cd2f9e3c5791e82fb7e6b91556cc3bd9e7691db4cc1a1beb0"}
Jul 27 20:35:28.829347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.829322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"affa9b0775e526f77913651a8eb11738e0a5620eeb7674ecebc4ab0ff53faffd"}
Jul 27 20:35:28.829472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.829384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"2270cf4eb9a2a55b6e5dc8b0c3ba01868e4ef8560cb856ee6ba3c0a1b0ae6ef5"}
Jul 27 20:35:28.830613 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.830577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wz462" event={"ID":"f7121382-96d9-460f-bfd9-bdfa44a6b64b","Type":"ContainerStarted","Data":"3f3a9ff1934ca7fd81c0d834a33325dd61a3c511c3a3a4def47eafbf66db4cbe"}
Jul 27 20:35:28.831800 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.831784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw" event={"ID":"0b33dce1-e944-4d01-b8df-854f98434262","Type":"ContainerStarted","Data":"056afb3105ad9fcfdf312dc0ba7240726ccb2531c694036d3f34bf930379e27c"}
Jul 27 20:35:28.833043 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.833020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8rgc" event={"ID":"120627ab-6ae5-48fc-9989-6de22a822e10","Type":"ContainerStarted","Data":"1e59d24889f1561b37fc90ceb4da9434a30ebda685b8f180b1a31944578e98c1"}
Jul 27 20:35:28.834370 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.834347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zj2jj" event={"ID":"0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2","Type":"ContainerStarted","Data":"a6e17b6fb70d99a525c9757f4ea5e768183e85d77697ca724329e5c19649b87e"}
Jul 27 20:35:28.835586 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.835564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2jbj4" event={"ID":"8a8bba74-5af3-4e93-bfdd-b8d35c760fa7","Type":"ContainerStarted","Data":"916e5f81afa86154c411a9f5ada383274ff8aa912d175498467eb79c6273da0a"}
Jul 27 20:35:28.836841 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.836822    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03067a3-a707-417d-b3b5-eeeea933c906" containerID="7e3dc19357cf1b4826d9088cffcb32b800ed5d144642ed626e92dfed5668cba1" exitCode=0
Jul 27 20:35:28.836918 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.836848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerDied","Data":"7e3dc19357cf1b4826d9088cffcb32b800ed5d144642ed626e92dfed5668cba1"}
Jul 27 20:35:28.841535 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.841499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6mrzj" podStartSLOduration=4.484964213 podStartE2EDuration="22.841489212s" totalImagesPullingTime="18.356524999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:28.841159996 +0000 UTC m=+22.721308700" watchObservedRunningTime="2026-07-27 20:35:28.841489212 +0000 UTC m=+22.721637917"
Jul 27 20:35:28.841780 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.841756    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-163.ec2.internal" podStartSLOduration=20.841750094 podStartE2EDuration="20.841750094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:11.803771469 +0000 UTC m=+5.683920178" watchObservedRunningTime="2026-07-27 20:35:28.841750094 +0000 UTC m=+22.721898802"
Jul 27 20:35:28.868069 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.868022    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zj2jj" podStartSLOduration=4.873050173 podStartE2EDuration="22.868008509s" totalImagesPullingTime="17.994958336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:28.855478783 +0000 UTC m=+22.735627488" watchObservedRunningTime="2026-07-27 20:35:28.868008509 +0000 UTC m=+22.748157214"
Jul 27 20:35:28.881395 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.881357    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wz462" podStartSLOduration=4.811772204 podStartE2EDuration="22.881345253s" totalImagesPullingTime="18.069573049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:28.881232851 +0000 UTC m=+22.761381560" watchObservedRunningTime="2026-07-27 20:35:28.881345253 +0000 UTC m=+22.761493958"
Jul 27 20:35:28.881489 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.881420    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2jbj4" podStartSLOduration=4.813293338 podStartE2EDuration="22.881415525s" totalImagesPullingTime="18.068122187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:28.867731782 +0000 UTC m=+22.747880487" watchObservedRunningTime="2026-07-27 20:35:28.881415525 +0000 UTC m=+22.761564234"
Jul 27 20:35:28.917138 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:28.917097    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8rgc" podStartSLOduration=4.843058992 podStartE2EDuration="22.917084268s" totalImagesPullingTime="18.074025276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:28.9170681 +0000 UTC m=+22.797216805" watchObservedRunningTime="2026-07-27 20:35:28.917084268 +0000 UTC m=+22.797232966"
Jul 27 20:35:29.199788 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.199763    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 20:35:29.584729 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.584569    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:35:29.19978507Z","UUID":"59bcd0c2-e47d-4c58-9f0c-93c898a77699","Handler":null,"Name":"","Endpoint":""}
Jul 27 20:35:29.586150 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.586130    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 20:35:29.586289 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.586166    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 20:35:29.843641 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.843593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:35:29.844391 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.844066    2581 generic.go:356] "Generic (PLEG): container finished" podID="f31ad677-ac5d-4f18-b038-3f6505a6dfa2" containerID="affa9b0775e526f77913651a8eb11738e0a5620eeb7674ecebc4ab0ff53faffd" exitCode=1
Jul 27 20:35:29.844391 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.844161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerDied","Data":"affa9b0775e526f77913651a8eb11738e0a5620eeb7674ecebc4ab0ff53faffd"}
Jul 27 20:35:29.844391 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.844288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"bdbeda5ccb76b303f71110e211f92768e2bdbc2a9377903c88a102377e387901"}
Jul 27 20:35:29.844391 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.844343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"b0bcdfca535182e65da572525b2dd60ba38da1ff6d72dfa440b29018d868e304"}
Jul 27 20:35:29.844391 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.844391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"0f2aedf836293b9aba237c7bc87c8365d5cbf8490de65d760c8ab4bec0f8543c"}
Jul 27 20:35:29.844691 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.844436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"565671c3ae2f5605dc3329bc5d88f2393235db846157b57fcc88cefa9eac1648"}
Jul 27 20:35:29.846803 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.846779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw" event={"ID":"0b33dce1-e944-4d01-b8df-854f98434262","Type":"ContainerStarted","Data":"92f07608573e773986a4d8ff6a2ffdfdedc4598412b888da8ac8362c5456bc43"}
Jul 27 20:35:29.848479 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.848455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bgzwj" event={"ID":"1d9bde2c-f800-4c98-8921-61e3847d82d8","Type":"ContainerStarted","Data":"d4d40325ee0faa7ccec13f351321c502b6cb18675cf8831ee5235f309d54ab11"}
Jul 27 20:35:29.866703 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:29.866646    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bgzwj" podStartSLOduration=5.865627743 podStartE2EDuration="23.866627404s" totalImagesPullingTime="18.000999661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:29.866048594 +0000 UTC m=+23.746197300" watchObservedRunningTime="2026-07-27 20:35:29.866627404 +0000 UTC m=+23.746776110"
Jul 27 20:35:30.728965 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:30.728928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:30.729125 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:30.728928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:30.729125 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:30.729072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:30.729125 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:30.729066    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:30.729304 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:30.729181    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:30.729361 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:30.729293    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:30.853270 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:30.853230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw" event={"ID":"0b33dce1-e944-4d01-b8df-854f98434262","Type":"ContainerStarted","Data":"8a2acaa785f2c2b920daf22f4e76b91cc531b3c5cc08cc96f6271130d712061e"}
Jul 27 20:35:30.888512 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:30.888466    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wttdw" podStartSLOduration=4.699686948 podStartE2EDuration="24.888450623s" totalImagesPullingTime="20.188763675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:30.888149975 +0000 UTC m=+24.768298716" watchObservedRunningTime="2026-07-27 20:35:30.888450623 +0000 UTC m=+24.768599332"
Jul 27 20:35:31.265349 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:31.265169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:31.265698 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:31.265679    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:31.856305 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:31.856277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:31.856785 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:31.856647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zj2jj"
Jul 27 20:35:32.728932 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:32.728904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:32.729113 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:32.728906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:32.729113 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:32.729013    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:32.729217 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:32.729111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:32.729217 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:32.729151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:32.729322 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:32.729219    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:32.864098 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:32.864044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:35:32.864739 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:32.864708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"b80e2a97cd4941a87741151dd4aec3d82f415862cf2bcf79ad21d9ccbf1813b3"}
Jul 27 20:35:33.868405 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:33.868371    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03067a3-a707-417d-b3b5-eeeea933c906" containerID="6ba524e417c5e230a5103b11c66bfd114cd69916bf76817541fb18a56eb93599" exitCode=0
Jul 27 20:35:33.868928 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:33.868460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerDied","Data":"6ba524e417c5e230a5103b11c66bfd114cd69916bf76817541fb18a56eb93599"}
Jul 27 20:35:34.729064 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:34.728976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:34.729064 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:34.729006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:34.729290 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:34.728976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:34.729290 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:34.729115    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:34.729290 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:34.729204    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:34.729440 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:34.729293    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:34.872899 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:34.872871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerStarted","Data":"1b6856ef2f49e39f24b93a90a0d28b6ee073d36c72be8604dabb510caa847d36"}
Jul 27 20:35:35.878814 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.878630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:35:35.879256 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.879230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"6fc6b4be32e2b5c00aaa3d7863fd62593fe58dec6a877e754b55d23d06ffe79e"}
Jul 27 20:35:35.879375 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.879355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:35.879516 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.879416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:35.879750 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.879734    2581 scope.go:117] "RemoveContainer" containerID="affa9b0775e526f77913651a8eb11738e0a5620eeb7674ecebc4ab0ff53faffd"
Jul 27 20:35:35.881439 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.881418    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03067a3-a707-417d-b3b5-eeeea933c906" containerID="1b6856ef2f49e39f24b93a90a0d28b6ee073d36c72be8604dabb510caa847d36" exitCode=0
Jul 27 20:35:35.881513 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.881453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerDied","Data":"1b6856ef2f49e39f24b93a90a0d28b6ee073d36c72be8604dabb510caa847d36"}
Jul 27 20:35:35.896434 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:35.896413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:36.730265 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.730020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:36.730397 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.730067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:36.730397 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:36.730381    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:36.730467 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.730093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:36.730467 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:36.730415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:36.730563 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:36.730488    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:36.886530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.886495    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03067a3-a707-417d-b3b5-eeeea933c906" containerID="9baf2eeb305710a234c8aa136175fd234c9bffc982ffca51e07871844b998d5e" exitCode=0
Jul 27 20:35:36.887013 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.886586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerDied","Data":"9baf2eeb305710a234c8aa136175fd234c9bffc982ffca51e07871844b998d5e"}
Jul 27 20:35:36.890696 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.890678    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:35:36.891019 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.890999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" event={"ID":"f31ad677-ac5d-4f18-b038-3f6505a6dfa2","Type":"ContainerStarted","Data":"d75a9cb6f8b0f2d8b6cd9353e3c1bea884ab44e1e356f5564692bfe473120498"}
Jul 27 20:35:36.891282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.891260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:36.907080 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.907049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:35:36.922100 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.922068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qvwm"]
Jul 27 20:35:36.922247 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.922187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:36.922323 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:36.922306    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:36.926093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.926055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-htcjh"]
Jul 27 20:35:36.926226 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.926173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:36.926289 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:36.926273    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:36.926344 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.926292    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p4zdq"]
Jul 27 20:35:36.926406 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.926363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:36.926456 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:36.926440    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:36.937594 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:36.934300    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk" podStartSLOduration=12.198882105 podStartE2EDuration="30.934284035s" totalImagesPullingTime="18.73540193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:36.934003366 +0000 UTC m=+30.814152072" watchObservedRunningTime="2026-07-27 20:35:36.934284035 +0000 UTC m=+30.814432752"
Jul 27 20:35:38.732306 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:38.732279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:38.732306 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:38.732290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:38.732889 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:38.732395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:38.732889 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:38.732476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:38.732889 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:38.732577    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:38.732889 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:38.732661    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:40.274264 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.274225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:40.274876 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.274359    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:40.274876 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.274416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:40.274876 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.274426    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret podName:421f2241-7f4c-446b-ae1a-0fd24b7046d7 nodeName:}" failed. No retries permitted until 2026-07-27 20:36:12.274408408 +0000 UTC m=+66.154557110 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret") pod "global-pull-secret-syncer-p4zdq" (UID: "421f2241-7f4c-446b-ae1a-0fd24b7046d7") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:35:40.274876 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.274531    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:40.274876 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.274594    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs podName:f35da8cb-24bc-4c87-9821-df562697ed0c nodeName:}" failed. No retries permitted until 2026-07-27 20:36:12.27457979 +0000 UTC m=+66.154728489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs") pod "network-metrics-daemon-7qvwm" (UID: "f35da8cb-24bc-4c87-9821-df562697ed0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:35:40.375043 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.375005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:40.375219 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.375152    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:35:40.375219 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.375170    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:35:40.375219 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.375180    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vjhg2 for pod openshift-network-diagnostics/network-check-target-htcjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:40.375340 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.375231    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2 podName:1f39d1d7-d09a-4a99-ad3d-eeb603597991 nodeName:}" failed. No retries permitted until 2026-07-27 20:36:12.375217466 +0000 UTC m=+66.255366150 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjhg2" (UniqueName: "kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2") pod "network-check-target-htcjh" (UID: "1f39d1d7-d09a-4a99-ad3d-eeb603597991") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:35:40.731465 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.731431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:40.731465 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.731467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:40.731724 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.731539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:40.731724 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.731567    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qvwm" podUID="f35da8cb-24bc-4c87-9821-df562697ed0c"
Jul 27 20:35:40.731724 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.731637    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htcjh" podUID="1f39d1d7-d09a-4a99-ad3d-eeb603597991"
Jul 27 20:35:40.731724 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:40.731711    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p4zdq" podUID="421f2241-7f4c-446b-ae1a-0fd24b7046d7"
Jul 27 20:35:40.930081 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.929997    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-163.ec2.internal" event="NodeReady"
Jul 27 20:35:40.930946 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.930602    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 20:35:40.975273 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.975241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-78w82"]
Jul 27 20:35:40.993665 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.993633    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wr22r"]
Jul 27 20:35:40.993845 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.993781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:40.996182 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.996141    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 20:35:40.996416 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.996141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 20:35:40.996501 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:40.996163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgqwx\""
Jul 27 20:35:41.017985 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.017955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78w82"]
Jul 27 20:35:41.018142 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.018002    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wr22r"]
Jul 27 20:35:41.018142 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.018025    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6j6m8"]
Jul 27 20:35:41.018254 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.018155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.020270 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.020249    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8t4xf\""
Jul 27 20:35:41.020513 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.020495    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 20:35:41.020620 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.020496    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 20:35:41.020620 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.020588    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 20:35:41.020736 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.020590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 20:35:41.040443 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.040415    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6j6m8"]
Jul 27 20:35:41.040580 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.040528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.044202 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.044175    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 20:35:41.044438 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.044418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mlcnn\""
Jul 27 20:35:41.044532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.044423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 20:35:41.044602 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.044486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 20:35:41.080354 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.080320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c05484d8-2dce-4067-b142-98c20367d823-config-volume\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.080505 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.080376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtl5\" (UniqueName: \"kubernetes.io/projected/c05484d8-2dce-4067-b142-98c20367d823-kube-api-access-2dtl5\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.080505 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.080392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05484d8-2dce-4067-b142-98c20367d823-tmp-dir\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.080505 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.080490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c05484d8-2dce-4067-b142-98c20367d823-metrics-tls\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.181833 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccdaf42f-295b-4bef-8887-f1025684914f-crio-socket\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.181833 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccdaf42f-295b-4bef-8887-f1025684914f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.181833 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c05484d8-2dce-4067-b142-98c20367d823-metrics-tls\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.182049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qnl\" (UniqueName: \"kubernetes.io/projected/ccdaf42f-295b-4bef-8887-f1025684914f-kube-api-access-72qnl\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.182049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccdaf42f-295b-4bef-8887-f1025684914f-data-volume\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.182049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da1c51f1-5464-4671-aa71-fa506416e311-cert\") pod \"ingress-canary-6j6m8\" (UID: \"da1c51f1-5464-4671-aa71-fa506416e311\") " pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.182049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c05484d8-2dce-4067-b142-98c20367d823-config-volume\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.182049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.181989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtl5\" (UniqueName: \"kubernetes.io/projected/c05484d8-2dce-4067-b142-98c20367d823-kube-api-access-2dtl5\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.182210 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.182091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05484d8-2dce-4067-b142-98c20367d823-tmp-dir\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.182210 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.182109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccdaf42f-295b-4bef-8887-f1025684914f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.182210 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.182129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fv4h\" (UniqueName: \"kubernetes.io/projected/da1c51f1-5464-4671-aa71-fa506416e311-kube-api-access-5fv4h\") pod \"ingress-canary-6j6m8\" (UID: \"da1c51f1-5464-4671-aa71-fa506416e311\") " pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.182371 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.182355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c05484d8-2dce-4067-b142-98c20367d823-tmp-dir\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.182537 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.182515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c05484d8-2dce-4067-b142-98c20367d823-config-volume\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.185999 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.185966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c05484d8-2dce-4067-b142-98c20367d823-metrics-tls\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.189893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.189869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtl5\" (UniqueName: \"kubernetes.io/projected/c05484d8-2dce-4067-b142-98c20367d823-kube-api-access-2dtl5\") pod \"dns-default-78w82\" (UID: \"c05484d8-2dce-4067-b142-98c20367d823\") " pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.283140 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccdaf42f-295b-4bef-8887-f1025684914f-crio-socket\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283140 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccdaf42f-295b-4bef-8887-f1025684914f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qnl\" (UniqueName: \"kubernetes.io/projected/ccdaf42f-295b-4bef-8887-f1025684914f-kube-api-access-72qnl\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccdaf42f-295b-4bef-8887-f1025684914f-data-volume\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da1c51f1-5464-4671-aa71-fa506416e311-cert\") pod \"ingress-canary-6j6m8\" (UID: \"da1c51f1-5464-4671-aa71-fa506416e311\") " pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccdaf42f-295b-4bef-8887-f1025684914f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fv4h\" (UniqueName: \"kubernetes.io/projected/da1c51f1-5464-4671-aa71-fa506416e311-kube-api-access-5fv4h\") pod \"ingress-canary-6j6m8\" (UID: \"da1c51f1-5464-4671-aa71-fa506416e311\") " pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccdaf42f-295b-4bef-8887-f1025684914f-crio-socket\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccdaf42f-295b-4bef-8887-f1025684914f-data-volume\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.283885 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.283808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccdaf42f-295b-4bef-8887-f1025684914f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.285712 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.285694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccdaf42f-295b-4bef-8887-f1025684914f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.285827 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.285801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da1c51f1-5464-4671-aa71-fa506416e311-cert\") pod \"ingress-canary-6j6m8\" (UID: \"da1c51f1-5464-4671-aa71-fa506416e311\") " pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.296954 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.296923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qnl\" (UniqueName: \"kubernetes.io/projected/ccdaf42f-295b-4bef-8887-f1025684914f-kube-api-access-72qnl\") pod \"insights-runtime-extractor-wr22r\" (UID: \"ccdaf42f-295b-4bef-8887-f1025684914f\") " pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.297076 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.297062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fv4h\" (UniqueName: \"kubernetes.io/projected/da1c51f1-5464-4671-aa71-fa506416e311-kube-api-access-5fv4h\") pod \"ingress-canary-6j6m8\" (UID: \"da1c51f1-5464-4671-aa71-fa506416e311\") " pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.303928 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.303907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:41.326241 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.326224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wr22r"
Jul 27 20:35:41.349323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.349299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6j6m8"
Jul 27 20:35:41.583176 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.583013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6j6m8"]
Jul 27 20:35:41.587407 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:41.587382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1c51f1_5464_4671_aa71_fa506416e311.slice/crio-833d43fcff32b1a3f72cf676f772e2ba60c3be863aa62d2531e4024b4ab72e7a WatchSource:0}: Error finding container 833d43fcff32b1a3f72cf676f772e2ba60c3be863aa62d2531e4024b4ab72e7a: Status 404 returned error can't find the container with id 833d43fcff32b1a3f72cf676f772e2ba60c3be863aa62d2531e4024b4ab72e7a
Jul 27 20:35:41.590854 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.590830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wr22r"]
Jul 27 20:35:41.594573 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:41.594083    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdaf42f_295b_4bef_8887_f1025684914f.slice/crio-442c6cf6e71866fad7773759bafa6b95fb8a128f582beab05ea61511c9b8a5c7 WatchSource:0}: Error finding container 442c6cf6e71866fad7773759bafa6b95fb8a128f582beab05ea61511c9b8a5c7: Status 404 returned error can't find the container with id 442c6cf6e71866fad7773759bafa6b95fb8a128f582beab05ea61511c9b8a5c7
Jul 27 20:35:41.594830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.594806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78w82"]
Jul 27 20:35:41.597588 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:41.597560    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05484d8_2dce_4067_b142_98c20367d823.slice/crio-0144c454a88e60115e8bc850789b4bca59f51d5b6451c10f5e2078bc603b47c8 WatchSource:0}: Error finding container 0144c454a88e60115e8bc850789b4bca59f51d5b6451c10f5e2078bc603b47c8: Status 404 returned error can't find the container with id 0144c454a88e60115e8bc850789b4bca59f51d5b6451c10f5e2078bc603b47c8
Jul 27 20:35:41.903788 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.903751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wr22r" event={"ID":"ccdaf42f-295b-4bef-8887-f1025684914f","Type":"ContainerStarted","Data":"442c6cf6e71866fad7773759bafa6b95fb8a128f582beab05ea61511c9b8a5c7"}
Jul 27 20:35:41.904864 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.904835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6j6m8" event={"ID":"da1c51f1-5464-4671-aa71-fa506416e311","Type":"ContainerStarted","Data":"833d43fcff32b1a3f72cf676f772e2ba60c3be863aa62d2531e4024b4ab72e7a"}
Jul 27 20:35:41.907394 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.907372    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03067a3-a707-417d-b3b5-eeeea933c906" containerID="0ee3ea7405f7721927125a630adb7bc4d2a6304337245acc871b8ceafb72b633" exitCode=0
Jul 27 20:35:41.907499 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.907440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerDied","Data":"0ee3ea7405f7721927125a630adb7bc4d2a6304337245acc871b8ceafb72b633"}
Jul 27 20:35:41.908528 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:41.908495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78w82" event={"ID":"c05484d8-2dce-4067-b142-98c20367d823","Type":"ContainerStarted","Data":"0144c454a88e60115e8bc850789b4bca59f51d5b6451c10f5e2078bc603b47c8"}
Jul 27 20:35:42.728530 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.728498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:35:42.729188 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.728497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:35:42.729188 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.728706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:35:42.735748 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.735722    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:35:42.735980 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.735962    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:35:42.736217 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.736203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:35:42.736405 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.736387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqrvq\""
Jul 27 20:35:42.736611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.736595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pl2wx\""
Jul 27 20:35:42.736826 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.736812    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:35:42.918428 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.918385    2581 generic.go:356] "Generic (PLEG): container finished" podID="b03067a3-a707-417d-b3b5-eeeea933c906" containerID="d3c3dae6e9d24bf983d4003d05146b65343b1ee6a532cbd80f4acb21e47344cd" exitCode=0
Jul 27 20:35:42.918631 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.918442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerDied","Data":"d3c3dae6e9d24bf983d4003d05146b65343b1ee6a532cbd80f4acb21e47344cd"}
Jul 27 20:35:42.943194 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.943162    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c88b4d96c-44wj6"]
Jul 27 20:35:42.956316 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.956054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c88b4d96c-44wj6"]
Jul 27 20:35:42.956316 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.956184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:42.961656 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.961370    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:35:42.961656 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.961649    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:35:42.961893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.961872    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:35:42.962201 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.962181    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:35:42.962437 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.962411    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:35:42.962940 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.962921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:35:42.963234 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.963215    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wp4fx\""
Jul 27 20:35:42.963308 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:42.963264    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:35:43.099323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.099055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-oauth-serving-cert\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.099323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.099107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-oauth-config\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.099323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.099134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-service-ca\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.099323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.099208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-config\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.099323 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.099237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwhc\" (UniqueName: \"kubernetes.io/projected/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-kube-api-access-pfwhc\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.099736 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.099340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-serving-cert\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.200487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.200245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-config\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.200487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.200296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwhc\" (UniqueName: \"kubernetes.io/projected/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-kube-api-access-pfwhc\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.200487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.200355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-serving-cert\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.200487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.200401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-oauth-serving-cert\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.200487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.200431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-oauth-config\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.200487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.200454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-service-ca\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.201119 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.201087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-config\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.201214 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.201161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-service-ca\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.201214 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.201192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-oauth-serving-cert\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.207029 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.206860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-serving-cert\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.207192 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.206867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-oauth-config\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.210058 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.210030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwhc\" (UniqueName: \"kubernetes.io/projected/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-kube-api-access-pfwhc\") pod \"console-7c88b4d96c-44wj6\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") " pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.279710 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.279627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:35:43.925678 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.925609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" event={"ID":"b03067a3-a707-417d-b3b5-eeeea933c906","Type":"ContainerStarted","Data":"e2d7b81c7d620bf6c49c693220fad1eaec56746ee73483a814237c2bb0eb9215"}
Jul 27 20:35:43.947628 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:43.947299    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hjlgd" podStartSLOduration=9.732086694 podStartE2EDuration="37.947276339s" totalImagesPullingTime="28.215189645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:43.946438605 +0000 UTC m=+37.826587311" watchObservedRunningTime="2026-07-27 20:35:43.947276339 +0000 UTC m=+37.827425049"
Jul 27 20:35:47.991728 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:47.991642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c88b4d96c-44wj6"]
Jul 27 20:35:48.309524 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:48.309485    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf461c1eb_3ca6_438d_bf24_eddbcc7b1037.slice/crio-13287ce3de6289697409fd0bc21c695b2c1e9d4a193aa3e3592c98681f310a5b WatchSource:0}: Error finding container 13287ce3de6289697409fd0bc21c695b2c1e9d4a193aa3e3592c98681f310a5b: Status 404 returned error can't find the container with id 13287ce3de6289697409fd0bc21c695b2c1e9d4a193aa3e3592c98681f310a5b
Jul 27 20:35:48.942440 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.942395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wr22r" event={"ID":"ccdaf42f-295b-4bef-8887-f1025684914f","Type":"ContainerStarted","Data":"28172e02bbd436399d684621fad22df9cb24ca88b243cf58f404cbccec7d3c10"}
Jul 27 20:35:48.943921 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.943885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88b4d96c-44wj6" event={"ID":"f461c1eb-3ca6-438d-bf24-eddbcc7b1037","Type":"ContainerStarted","Data":"13287ce3de6289697409fd0bc21c695b2c1e9d4a193aa3e3592c98681f310a5b"}
Jul 27 20:35:48.945490 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.945466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6j6m8" event={"ID":"da1c51f1-5464-4671-aa71-fa506416e311","Type":"ContainerStarted","Data":"a94d54287b8a50607bff42976e4455b0479ba5a321739006d7b9b21bfe67e406"}
Jul 27 20:35:48.947453 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.947432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78w82" event={"ID":"c05484d8-2dce-4067-b142-98c20367d823","Type":"ContainerStarted","Data":"f115c957afda84d4e220f007fd06e494f917c102df846644b77fcf419478a3e4"}
Jul 27 20:35:48.947589 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.947475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78w82" event={"ID":"c05484d8-2dce-4067-b142-98c20367d823","Type":"ContainerStarted","Data":"7edf983c2855d7f09f6b8349003a946f61d5f4991d6b479090e950950ab44e8b"}
Jul 27 20:35:48.947699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.947673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:48.963450 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:48.963375    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6j6m8" podStartSLOduration=2.248259019 podStartE2EDuration="8.963362692s" totalImagesPullingTime="6.715103673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:48.962673883 +0000 UTC m=+42.842822589" watchObservedRunningTime="2026-07-27 20:35:48.963362692 +0000 UTC m=+42.843511375"
Jul 27 20:35:49.935766 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:49.935618    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-78w82" podStartSLOduration=3.227632662 podStartE2EDuration="9.935591096s" totalImagesPullingTime="6.707958434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:48.979046617 +0000 UTC m=+42.859195324" watchObservedRunningTime="2026-07-27 20:35:49.935591096 +0000 UTC m=+43.815739802"
Jul 27 20:35:49.937298 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:49.936335    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f7d96b6b9-bzjs4"]
Jul 27 20:35:49.949574 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:49.949483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:49.951458 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:49.951435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7d96b6b9-bzjs4"]
Jul 27 20:35:49.959573 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:49.959082    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 20:35:50.055825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-serving-cert\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.055825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-console-config\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.055825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-trusted-ca-bundle\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.055825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg49g\" (UniqueName: \"kubernetes.io/projected/ad70af87-bb93-45de-ba89-a743143368b9-kube-api-access-pg49g\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.055825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-oauth-serving-cert\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.055825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-oauth-config\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.056269 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.055855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-service-ca\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.156994 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.156958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg49g\" (UniqueName: \"kubernetes.io/projected/ad70af87-bb93-45de-ba89-a743143368b9-kube-api-access-pg49g\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.157160 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.157028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-oauth-serving-cert\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.157160 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.157066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-oauth-config\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.157160 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.157109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-service-ca\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.157160 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.157144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-serving-cert\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.157362 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.157168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-console-config\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.157362 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.157190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-trusted-ca-bundle\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.158215 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.158166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-service-ca\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.158912 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.158859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-console-config\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.159799 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.159637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-trusted-ca-bundle\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.160698 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.160673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-oauth-serving-cert\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.163283 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.163260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-oauth-config\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.164002 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.163959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-serving-cert\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.166726 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.166686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg49g\" (UniqueName: \"kubernetes.io/projected/ad70af87-bb93-45de-ba89-a743143368b9-kube-api-access-pg49g\") pod \"console-f7d96b6b9-bzjs4\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") " pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.266608 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.266349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:35:50.444244 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.444200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7d96b6b9-bzjs4"]
Jul 27 20:35:50.448181 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:50.448154    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad70af87_bb93_45de_ba89_a743143368b9.slice/crio-5e2d72c41a4f76097c823fb855f331369f0143f5c11a55eb86931be3fd7d21e3 WatchSource:0}: Error finding container 5e2d72c41a4f76097c823fb855f331369f0143f5c11a55eb86931be3fd7d21e3: Status 404 returned error can't find the container with id 5e2d72c41a4f76097c823fb855f331369f0143f5c11a55eb86931be3fd7d21e3
Jul 27 20:35:50.954152 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:50.954114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7d96b6b9-bzjs4" event={"ID":"ad70af87-bb93-45de-ba89-a743143368b9","Type":"ContainerStarted","Data":"5e2d72c41a4f76097c823fb855f331369f0143f5c11a55eb86931be3fd7d21e3"}
Jul 27 20:35:52.178112 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.178083    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"]
Jul 27 20:35:52.223053 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.223018    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"]
Jul 27 20:35:52.223208 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.223126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.225814 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.225792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 20:35:52.225814 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.225812    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 20:35:52.226095 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.226072    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 20:35:52.226589 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.226574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 20:35:52.226861 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.226836    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 20:35:52.226970 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.226910    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xp26w\""
Jul 27 20:35:52.378740 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.378698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.378931 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.378800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.378931 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.378842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpn9h\" (UniqueName: \"kubernetes.io/projected/f4104fab-b36f-4592-8605-e558db135fe6-kube-api-access-kpn9h\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.379026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.378928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4104fab-b36f-4592-8605-e558db135fe6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.480223 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.480143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.480223 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.480201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpn9h\" (UniqueName: \"kubernetes.io/projected/f4104fab-b36f-4592-8605-e558db135fe6-kube-api-access-kpn9h\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.480223 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.480232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4104fab-b36f-4592-8605-e558db135fe6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.480522 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.480314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.480522 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:52.480448    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 20:35:52.480522 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:35:52.480511    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-tls podName:f4104fab-b36f-4592-8605-e558db135fe6 nodeName:}" failed. No retries permitted until 2026-07-27 20:35:52.980493216 +0000 UTC m=+46.860641904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-cnwnr" (UID: "f4104fab-b36f-4592-8605-e558db135fe6") : secret "prometheus-operator-tls" not found
Jul 27 20:35:52.481224 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.481196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4104fab-b36f-4592-8605-e558db135fe6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.482962 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.482940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.488896 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.488870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpn9h\" (UniqueName: \"kubernetes.io/projected/f4104fab-b36f-4592-8605-e558db135fe6-kube-api-access-kpn9h\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.984706 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.984668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:52.987036 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:52.987016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4104fab-b36f-4592-8605-e558db135fe6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cnwnr\" (UID: \"f4104fab-b36f-4592-8605-e558db135fe6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:53.134459 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:53.134428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"
Jul 27 20:35:54.351143 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:54.351114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cnwnr"]
Jul 27 20:35:54.354963 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:35:54.354933    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4104fab_b36f_4592_8605_e558db135fe6.slice/crio-7e3ac6634c5c036987223c74657ec9123187c23b45eb36446c5bcf00f3642281 WatchSource:0}: Error finding container 7e3ac6634c5c036987223c74657ec9123187c23b45eb36446c5bcf00f3642281: Status 404 returned error can't find the container with id 7e3ac6634c5c036987223c74657ec9123187c23b45eb36446c5bcf00f3642281
Jul 27 20:35:54.968509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:54.968462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wr22r" event={"ID":"ccdaf42f-295b-4bef-8887-f1025684914f","Type":"ContainerStarted","Data":"bd0edb21f7229c59bee2fe0b8501caec3ef90275b517d5370d4f6770927c69f9"}
Jul 27 20:35:54.970702 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:54.970673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88b4d96c-44wj6" event={"ID":"f461c1eb-3ca6-438d-bf24-eddbcc7b1037","Type":"ContainerStarted","Data":"24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3"}
Jul 27 20:35:54.972376 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:54.972350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7d96b6b9-bzjs4" event={"ID":"ad70af87-bb93-45de-ba89-a743143368b9","Type":"ContainerStarted","Data":"f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1"}
Jul 27 20:35:54.973573 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:54.973534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr" event={"ID":"f4104fab-b36f-4592-8605-e558db135fe6","Type":"ContainerStarted","Data":"7e3ac6634c5c036987223c74657ec9123187c23b45eb36446c5bcf00f3642281"}
Jul 27 20:35:54.987242 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:54.987186    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c88b4d96c-44wj6" podStartSLOduration=7.071287139 podStartE2EDuration="12.987168786s" totalImagesPullingTime="5.915881647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:54.987030191 +0000 UTC m=+48.867178898" watchObservedRunningTime="2026-07-27 20:35:54.987168786 +0000 UTC m=+48.867317493"
Jul 27 20:35:55.004461 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:55.004407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f7d96b6b9-bzjs4" podStartSLOduration=1.902455762 podStartE2EDuration="6.004391726s" totalImagesPullingTime="4.101935964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:55.003791256 +0000 UTC m=+48.883939960" watchObservedRunningTime="2026-07-27 20:35:55.004391726 +0000 UTC m=+48.884540409"
Jul 27 20:35:56.983099 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:56.983060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr" event={"ID":"f4104fab-b36f-4592-8605-e558db135fe6","Type":"ContainerStarted","Data":"f99e9077a031cc3691c56e039a470ce014cbd72eefa5ad3d30bbcdc2cadf6742"}
Jul 27 20:35:56.983591 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:56.983129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr" event={"ID":"f4104fab-b36f-4592-8605-e558db135fe6","Type":"ContainerStarted","Data":"07ba2d096be52884f49d53ca64da37d9d2af97ae18812bc8b65f1a85f50e7db5"}
Jul 27 20:35:56.999433 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:56.999374    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-cnwnr" podStartSLOduration=3.316575954 podStartE2EDuration="4.999357819s" totalImagesPullingTime="1.682781865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:56.99830155 +0000 UTC m=+50.878450262" watchObservedRunningTime="2026-07-27 20:35:56.999357819 +0000 UTC m=+50.879506523"
Jul 27 20:35:58.954305 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:58.954272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-78w82"
Jul 27 20:35:58.992472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:58.992442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wr22r" event={"ID":"ccdaf42f-295b-4bef-8887-f1025684914f","Type":"ContainerStarted","Data":"add959ab6bf55636011df07442142f97ed97a4135a56d4e75ea3c99ef12037ee"}
Jul 27 20:35:59.014707 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:35:59.014657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wr22r" podStartSLOduration=2.266776631 podStartE2EDuration="19.0146404s" totalImagesPullingTime="16.747863769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:59.013843563 +0000 UTC m=+52.893992286" watchObservedRunningTime="2026-07-27 20:35:59.0146404 +0000 UTC m=+52.894789105"
Jul 27 20:36:00.267122 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:00.267083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:36:00.267122 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:00.267135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:36:00.271898 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:00.271878    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:36:01.002248 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:01.002221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:36:01.046072 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:01.046031    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c88b4d96c-44wj6"]
Jul 27 20:36:03.205181 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.205149    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"]
Jul 27 20:36:03.207841 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.207823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.210357 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.210327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rjmgg\""
Jul 27 20:36:03.210357 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.210327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 20:36:03.210514 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.210368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 20:36:03.218092 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.218070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"]
Jul 27 20:36:03.250091 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.250067    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-npbcl"]
Jul 27 20:36:03.252195 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.252180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.254504 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.254482    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 20:36:03.254621 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.254529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 20:36:03.254621 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.254492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dfz44\""
Jul 27 20:36:03.254621 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.254493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 20:36:03.264524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.264648 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd3382e-56b6-4756-829a-0caee9d5303d-metrics-client-ca\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264648 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264648 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4w2q\" (UniqueName: \"kubernetes.io/projected/8fd3382e-56b6-4756-829a-0caee9d5303d-kube-api-access-s4w2q\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264808 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/5724c33a-c789-4c9e-bafc-bae2d1b073cd-kube-api-access-jz25x\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.264808 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-textfile\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264808 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-wtmp\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264808 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-accelerators-collector-config\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264957 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-sys\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264957 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-tls\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.264957 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5724c33a-c789-4c9e-bafc-bae2d1b073cd-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.265058 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.264987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-root\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.265058 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.265011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.280431 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.280408    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:36:03.365940 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.365905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/5724c33a-c789-4c9e-bafc-bae2d1b073cd-kube-api-access-jz25x\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.365940 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.365944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-textfile\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-wtmp\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-accelerators-collector-config\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-sys\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-tls\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5724c33a-c789-4c9e-bafc-bae2d1b073cd-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-wtmp\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-sys\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366184 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-root\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd3382e-56b6-4756-829a-0caee9d5303d-root\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd3382e-56b6-4756-829a-0caee9d5303d-metrics-client-ca\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-textfile\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:03.366366    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4w2q\" (UniqueName: \"kubernetes.io/projected/8fd3382e-56b6-4756-829a-0caee9d5303d-kube-api-access-s4w2q\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366611 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:03.366432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-tls podName:5724c33a-c789-4c9e-bafc-bae2d1b073cd nodeName:}" failed. No retries permitted until 2026-07-27 20:36:03.866409769 +0000 UTC m=+57.746558458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-9wctv" (UID: "5724c33a-c789-4c9e-bafc-bae2d1b073cd") : secret "openshift-state-metrics-tls" not found
Jul 27 20:36:03.366972 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd3382e-56b6-4756-829a-0caee9d5303d-metrics-client-ca\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.366972 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5724c33a-c789-4c9e-bafc-bae2d1b073cd-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.367074 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.366962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-accelerators-collector-config\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.368767 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.368742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.368864 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.368767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd3382e-56b6-4756-829a-0caee9d5303d-node-exporter-tls\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.368864 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.368778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.376743 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.376715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4w2q\" (UniqueName: \"kubernetes.io/projected/8fd3382e-56b6-4756-829a-0caee9d5303d-kube-api-access-s4w2q\") pod \"node-exporter-npbcl\" (UID: \"8fd3382e-56b6-4756-829a-0caee9d5303d\") " pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.376948 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.376932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz25x\" (UniqueName: \"kubernetes.io/projected/5724c33a-c789-4c9e-bafc-bae2d1b073cd-kube-api-access-jz25x\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.560514 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.560415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-npbcl"
Jul 27 20:36:03.569764 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:03.569740    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd3382e_56b6_4756_829a_0caee9d5303d.slice/crio-21e0ea97777b4b55ef60e255d48eced2e4fcfe1e803ffbfeb504ea12577ba1ce WatchSource:0}: Error finding container 21e0ea97777b4b55ef60e255d48eced2e4fcfe1e803ffbfeb504ea12577ba1ce: Status 404 returned error can't find the container with id 21e0ea97777b4b55ef60e255d48eced2e4fcfe1e803ffbfeb504ea12577ba1ce
Jul 27 20:36:03.870208 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.870180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:03.872454 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:03.872434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5724c33a-c789-4c9e-bafc-bae2d1b073cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9wctv\" (UID: \"5724c33a-c789-4c9e-bafc-bae2d1b073cd\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:04.008567 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.008520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npbcl" event={"ID":"8fd3382e-56b6-4756-829a-0caee9d5303d","Type":"ContainerStarted","Data":"21e0ea97777b4b55ef60e255d48eced2e4fcfe1e803ffbfeb504ea12577ba1ce"}
Jul 27 20:36:04.116391 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.116353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"
Jul 27 20:36:04.263913 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.263869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv"]
Jul 27 20:36:04.268117 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:04.268085    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5724c33a_c789_4c9e_bafc_bae2d1b073cd.slice/crio-3b5088be1771b7543028c912eac25c4d9cbb09ed9c5d3cb5c75d3b51b9432276 WatchSource:0}: Error finding container 3b5088be1771b7543028c912eac25c4d9cbb09ed9c5d3cb5c75d3b51b9432276: Status 404 returned error can't find the container with id 3b5088be1771b7543028c912eac25c4d9cbb09ed9c5d3cb5c75d3b51b9432276
Jul 27 20:36:04.294198 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.294175    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:36:04.297333 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.297315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.302512 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 20:36:04.302711 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 20:36:04.302793 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 20:36:04.303163 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 20:36:04.303163 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 20:36:04.303163 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.303008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 20:36:04.303163 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 20:36:04.303163 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.302687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b9jp9\""
Jul 27 20:36:04.303414 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.303233    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 20:36:04.310966 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.310947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 20:36:04.312660 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.312641    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:36:04.375470 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375617 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ac4e19-b443-49fa-8a24-3e010586a5a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375617 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ac4e19-b443-49fa-8a24-3e010586a5a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375726 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5ac4e19-b443-49fa-8a24-3e010586a5a8-config-out\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375726 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375726 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375863 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375863 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5ac4e19-b443-49fa-8a24-3e010586a5a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375925 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-web-config\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.375925 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.376026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.376026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.375951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5ac4e19-b443-49fa-8a24-3e010586a5a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.376125 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.376030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj699\" (UniqueName: \"kubernetes.io/projected/b5ac4e19-b443-49fa-8a24-3e010586a5a8-kube-api-access-bj699\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.477501 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5ac4e19-b443-49fa-8a24-3e010586a5a8-config-out\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.477699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.477699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.477699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.477699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5ac4e19-b443-49fa-8a24-3e010586a5a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.477699 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:04.477665    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:04.477733    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-main-tls podName:b5ac4e19-b443-49fa-8a24-3e010586a5a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:36:04.977712771 +0000 UTC m=+58.857861469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b5ac4e19-b443-49fa-8a24-3e010586a5a8") : secret "alertmanager-main-tls" not found
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-web-config\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5ac4e19-b443-49fa-8a24-3e010586a5a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.477965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj699\" (UniqueName: \"kubernetes.io/projected/b5ac4e19-b443-49fa-8a24-3e010586a5a8-kube-api-access-bj699\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478021 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.478004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478366 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.478029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ac4e19-b443-49fa-8a24-3e010586a5a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.478366 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.478055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ac4e19-b443-49fa-8a24-3e010586a5a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.479160 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.478815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ac4e19-b443-49fa-8a24-3e010586a5a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.479160 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.478968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5ac4e19-b443-49fa-8a24-3e010586a5a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.479819 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.479794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5ac4e19-b443-49fa-8a24-3e010586a5a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.480802 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.480771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5ac4e19-b443-49fa-8a24-3e010586a5a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.480946 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.480921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.481542 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.481323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-web-config\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.481542 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.481495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5ac4e19-b443-49fa-8a24-3e010586a5a8-config-out\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.481914 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.481891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.482495 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.482471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.482616 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.482510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.483113 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.483094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.486808 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.486787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj699\" (UniqueName: \"kubernetes.io/projected/b5ac4e19-b443-49fa-8a24-3e010586a5a8-kube-api-access-bj699\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.982729 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.982651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:04.985127 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:04.985102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5ac4e19-b443-49fa-8a24-3e010586a5a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5ac4e19-b443-49fa-8a24-3e010586a5a8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:05.015652 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.015600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv" event={"ID":"5724c33a-c789-4c9e-bafc-bae2d1b073cd","Type":"ContainerStarted","Data":"c33c15d7442ef19bcb4d3c527588d1f4d30f4979b8dae866d62f5bf2167e90ef"}
Jul 27 20:36:05.015828 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.015685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv" event={"ID":"5724c33a-c789-4c9e-bafc-bae2d1b073cd","Type":"ContainerStarted","Data":"23d3a296e36038f6fc91c85c505d838b7fdd6cd6db9f822963bc0b450220d285"}
Jul 27 20:36:05.015828 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.015714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv" event={"ID":"5724c33a-c789-4c9e-bafc-bae2d1b073cd","Type":"ContainerStarted","Data":"3b5088be1771b7543028c912eac25c4d9cbb09ed9c5d3cb5c75d3b51b9432276"}
Jul 27 20:36:05.017441 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.017411    2581 generic.go:356] "Generic (PLEG): container finished" podID="8fd3382e-56b6-4756-829a-0caee9d5303d" containerID="86a6d5508c17f77e0d8fce87ed6a66143e0012c324874bbe1427824720637bdf" exitCode=0
Jul 27 20:36:05.017597 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.017468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npbcl" event={"ID":"8fd3382e-56b6-4756-829a-0caee9d5303d","Type":"ContainerDied","Data":"86a6d5508c17f77e0d8fce87ed6a66143e0012c324874bbe1427824720637bdf"}
Jul 27 20:36:05.218435 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.218411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:36:05.366587 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:05.366542    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:36:05.500450 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:05.500380    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ac4e19_b443_49fa_8a24_3e010586a5a8.slice/crio-686edc25a4146c85690f8419d1663e10ae7e1e5695d39eade487c07a057bb9a6 WatchSource:0}: Error finding container 686edc25a4146c85690f8419d1663e10ae7e1e5695d39eade487c07a057bb9a6: Status 404 returned error can't find the container with id 686edc25a4146c85690f8419d1663e10ae7e1e5695d39eade487c07a057bb9a6
Jul 27 20:36:06.026679 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:06.026588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npbcl" event={"ID":"8fd3382e-56b6-4756-829a-0caee9d5303d","Type":"ContainerStarted","Data":"d37041be37b7a6c46bc498e36425cebe0e82eb65cb417de3d62cfdc5f95005da"}
Jul 27 20:36:06.026679 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:06.026668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npbcl" event={"ID":"8fd3382e-56b6-4756-829a-0caee9d5303d","Type":"ContainerStarted","Data":"cd980e34b316bd9d14843ab795e6ca7ef194b63ae5c22f9857b435e4d36ffdf1"}
Jul 27 20:36:06.028108 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:06.028077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"686edc25a4146c85690f8419d1663e10ae7e1e5695d39eade487c07a057bb9a6"}
Jul 27 20:36:06.034002 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:06.031589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv" event={"ID":"5724c33a-c789-4c9e-bafc-bae2d1b073cd","Type":"ContainerStarted","Data":"be123b2bce4046cb303348cdc45ad8fe868c52bc1863ca5df1d9249bfbf7bacb"}
Jul 27 20:36:06.047653 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:06.047571    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-npbcl" podStartSLOduration=1.950658029 podStartE2EDuration="3.047541442s" totalImagesPullingTime="1.032959989s" totalInitContainerRuntime="63.923424ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:06.046706041 +0000 UTC m=+59.926854764" watchObservedRunningTime="2026-07-27 20:36:06.047541442 +0000 UTC m=+59.927690147"
Jul 27 20:36:06.063857 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:06.063806    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9wctv" podStartSLOduration=1.724840603 podStartE2EDuration="3.063785488s" totalImagesPullingTime="1.338944885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:06.062182448 +0000 UTC m=+59.942331154" watchObservedRunningTime="2026-07-27 20:36:06.063785488 +0000 UTC m=+59.943934173"
Jul 27 20:36:07.036145 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:07.036060    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5ac4e19-b443-49fa-8a24-3e010586a5a8" containerID="2b93c077df5fed42b73c7f7970de8abc51ffb8609b28b36cb1429174822dce13" exitCode=0
Jul 27 20:36:07.036492 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:07.036148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerDied","Data":"2b93c077df5fed42b73c7f7970de8abc51ffb8609b28b36cb1429174822dce13"}
Jul 27 20:36:08.012976 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.012945    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c84fd48f7-7hq6h"]
Jul 27 20:36:08.015098 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.015081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.027526 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.027494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c84fd48f7-7hq6h"]
Jul 27 20:36:08.107624 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-oauth-serving-cert\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.107624 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-service-ca\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.108077 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-console-config\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.108077 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-serving-cert\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.108077 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-oauth-config\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.108077 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-trusted-ca-bundle\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.108077 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.107823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmfs\" (UniqueName: \"kubernetes.io/projected/77bd1a68-055a-4072-99e9-8b8237bf211d-kube-api-access-sxmfs\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.208710 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-oauth-config\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.208710 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-trusted-ca-bundle\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.208710 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmfs\" (UniqueName: \"kubernetes.io/projected/77bd1a68-055a-4072-99e9-8b8237bf211d-kube-api-access-sxmfs\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-oauth-serving-cert\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-service-ca\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-console-config\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.208901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-serving-cert\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.209590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-oauth-serving-cert\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209723 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.209664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-trusted-ca-bundle\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209772 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.209737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-service-ca\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.209831 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.209781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-console-config\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.211494 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.211473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-oauth-config\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.211679 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.211659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-serving-cert\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.216470 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.216450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmfs\" (UniqueName: \"kubernetes.io/projected/77bd1a68-055a-4072-99e9-8b8237bf211d-kube-api-access-sxmfs\") pod \"console-c84fd48f7-7hq6h\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") " pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.325407 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.325338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:08.423894 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.423815    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"]
Jul 27 20:36:08.427194 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.427172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.429413 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.429389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 20:36:08.429515 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.429488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 20:36:08.429790 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.429768    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gzhws\""
Jul 27 20:36:08.429879 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.429804    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 20:36:08.429879 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.429862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 20:36:08.430536 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.430047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 20:36:08.435110 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.435087    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 20:36:08.442471 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.442420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"]
Jul 27 20:36:08.462901 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.462877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c84fd48f7-7hq6h"]
Jul 27 20:36:08.468839 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:08.468798    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bd1a68_055a_4072_99e9_8b8237bf211d.slice/crio-5104649fa2fb39838649bc92adb436145533d42e479a826235e003e0445cfeab WatchSource:0}: Error finding container 5104649fa2fb39838649bc92adb436145533d42e479a826235e003e0445cfeab: Status 404 returned error can't find the container with id 5104649fa2fb39838649bc92adb436145533d42e479a826235e003e0445cfeab
Jul 27 20:36:08.511572 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-serving-certs-ca-bundle\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511674 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511674 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829wj\" (UniqueName: \"kubernetes.io/projected/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-kube-api-access-829wj\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511766 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-federate-client-tls\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511766 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-secret-telemeter-client\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511766 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-telemeter-client-tls\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511897 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.511897 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.511843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-metrics-client-ca\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612305 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612382 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-metrics-client-ca\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612444 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-serving-certs-ca-bundle\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612486 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612583 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829wj\" (UniqueName: \"kubernetes.io/projected/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-kube-api-access-829wj\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612583 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-federate-client-tls\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612657 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-secret-telemeter-client\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.612657 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.612615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-telemeter-client-tls\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.613130 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.613105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-metrics-client-ca\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.613190 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.613173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.613906 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.613884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-serving-certs-ca-bundle\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.615420 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.615399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-telemeter-client-tls\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.616189 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.616169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-secret-telemeter-client\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.616535 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.616514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.616639 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.616602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-federate-client-tls\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.620363 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.620337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-829wj\" (UniqueName: \"kubernetes.io/projected/f634b6f0-ffb0-4dc2-a34d-e972be0430b6-kube-api-access-829wj\") pod \"telemeter-client-6dc76f687f-fzp9j\" (UID: \"f634b6f0-ffb0-4dc2-a34d-e972be0430b6\") " pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.739783 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.739754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"
Jul 27 20:36:08.867249 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.867223    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc76f687f-fzp9j"]
Jul 27 20:36:08.870517 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:08.870493    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf634b6f0_ffb0_4dc2_a34d_e972be0430b6.slice/crio-713279b04215d040a75fa2eb4a6684e5e740a2793803fd57a19876140375feb2 WatchSource:0}: Error finding container 713279b04215d040a75fa2eb4a6684e5e740a2793803fd57a19876140375feb2: Status 404 returned error can't find the container with id 713279b04215d040a75fa2eb4a6684e5e740a2793803fd57a19876140375feb2
Jul 27 20:36:08.906333 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:08.906313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fq5fk"
Jul 27 20:36:09.046790 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.046749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j" event={"ID":"f634b6f0-ffb0-4dc2-a34d-e972be0430b6","Type":"ContainerStarted","Data":"713279b04215d040a75fa2eb4a6684e5e740a2793803fd57a19876140375feb2"}
Jul 27 20:36:09.050422 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.050388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"32aba11ffd6ae3224de8b64bf319593c45eb07e532dd07bb88f1f4d9c9aff446"}
Jul 27 20:36:09.050585 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.050467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"b313a77a072480f1a9c3b225c8491111a4b375c5a0812914537d4878c6306697"}
Jul 27 20:36:09.050585 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.050512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"a64ddd52190fd54efaf70768482e013518aa825e089507c98481af8ac1bd6a9b"}
Jul 27 20:36:09.050585 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.050574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"f9bda33c56dbc62fc0d9ff7e91076f9531e2859eeac4433fbec040bca6763599"}
Jul 27 20:36:09.050731 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.050617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"3909dff02d9c1a208a5a6676b8e57ec618ec4e6aae36a092d64b7bfc03f725c8"}
Jul 27 20:36:09.052162 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.052139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c84fd48f7-7hq6h" event={"ID":"77bd1a68-055a-4072-99e9-8b8237bf211d","Type":"ContainerStarted","Data":"699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa"}
Jul 27 20:36:09.052272 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.052194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c84fd48f7-7hq6h" event={"ID":"77bd1a68-055a-4072-99e9-8b8237bf211d","Type":"ContainerStarted","Data":"5104649fa2fb39838649bc92adb436145533d42e479a826235e003e0445cfeab"}
Jul 27 20:36:09.072174 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.072128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c84fd48f7-7hq6h" podStartSLOduration=2.072111082 podStartE2EDuration="2.072111082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:09.070727859 +0000 UTC m=+62.950876581" watchObservedRunningTime="2026-07-27 20:36:09.072111082 +0000 UTC m=+62.952259787"
Jul 27 20:36:09.413602 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.413570    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:36:09.416669 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.416640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.419051 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419032    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:36:09.419204 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419181    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:36:09.419276 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419213    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:36:09.419276 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419229    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:36:09.419375 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419366    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:36:09.419730 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419578    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:36:09.419730 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:36:09.419730 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:36:09.419939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:36:09.419939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:36:09.419939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9ftx\""
Jul 27 20:36:09.419939 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.419905    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:36:09.420135 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.420111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:36:09.420538 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.420380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7lp4jukvo4k79\""
Jul 27 20:36:09.420538 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.420436    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:36:09.420707 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.420696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:36:09.423954 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.423933    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:36:09.435694 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.435671    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:36:09.521918 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.521836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.521918 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.521891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51a9e039-0de1-42c6-bd59-3db18e32084a-config-out\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522128 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.521945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522128 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.521984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51a9e039-0de1-42c6-bd59-3db18e32084a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522128 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-web-config\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522128 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522128 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-config\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522301 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kjk\" (UniqueName: \"kubernetes.io/projected/51a9e039-0de1-42c6-bd59-3db18e32084a-kube-api-access-n7kjk\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.522487 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.522481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623786 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623968 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623968 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623968 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kjk\" (UniqueName: \"kubernetes.io/projected/51a9e039-0de1-42c6-bd59-3db18e32084a-kube-api-access-n7kjk\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623968 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623968 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.623968 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.623997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51a9e039-0de1-42c6-bd59-3db18e32084a-config-out\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51a9e039-0de1-42c6-bd59-3db18e32084a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-web-config\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-config\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624282 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624871 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624871 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624871 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624871 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.624871 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.625129 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.624979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.625235 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.625211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.627683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.627568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.627683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.627568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.627683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.627607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.628155 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.628120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.628388 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.628336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.628701 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.628675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a9e039-0de1-42c6-bd59-3db18e32084a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.629045 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.628999    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51a9e039-0de1-42c6-bd59-3db18e32084a-config-out\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.629213 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.629192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.629286 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.629257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.629726 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.629682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.630210 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.630187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-web-config\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.630503 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.630478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.630892 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.630842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51a9e039-0de1-42c6-bd59-3db18e32084a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.631455 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.631431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.631769 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.631752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kjk\" (UniqueName: \"kubernetes.io/projected/51a9e039-0de1-42c6-bd59-3db18e32084a-kube-api-access-n7kjk\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.632202 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.632182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/51a9e039-0de1-42c6-bd59-3db18e32084a-config\") pod \"prometheus-k8s-0\" (UID: \"51a9e039-0de1-42c6-bd59-3db18e32084a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.729243 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.729178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:09.894842 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:09.894808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:36:09.897013 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:09.896987    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a9e039_0de1_42c6_bd59_3db18e32084a.slice/crio-6eaf68177fb035223fe09dc21c175a8b9704847ce36fb75738bbe3c957a4804f WatchSource:0}: Error finding container 6eaf68177fb035223fe09dc21c175a8b9704847ce36fb75738bbe3c957a4804f: Status 404 returned error can't find the container with id 6eaf68177fb035223fe09dc21c175a8b9704847ce36fb75738bbe3c957a4804f
Jul 27 20:36:10.057671 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:10.057584    2581 generic.go:356] "Generic (PLEG): container finished" podID="51a9e039-0de1-42c6-bd59-3db18e32084a" containerID="4c872974f09f81b62abe6a3d5c5e5c9b85f600b64a2a742cc346a4607f04299f" exitCode=0
Jul 27 20:36:10.057836 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:10.057683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerDied","Data":"4c872974f09f81b62abe6a3d5c5e5c9b85f600b64a2a742cc346a4607f04299f"}
Jul 27 20:36:10.057836 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:10.057778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"6eaf68177fb035223fe09dc21c175a8b9704847ce36fb75738bbe3c957a4804f"}
Jul 27 20:36:10.062019 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:10.061988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5ac4e19-b443-49fa-8a24-3e010586a5a8","Type":"ContainerStarted","Data":"7596d88bdfabf7077658701b62fcf96e5b22d509bff628ef4eca52638f51fd57"}
Jul 27 20:36:10.108105 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:10.108051    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.971400177 podStartE2EDuration="6.10802965s" totalImagesPullingTime="3.121026956s" totalInitContainerRuntime="15.602517ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:10.106315762 +0000 UTC m=+63.986464468" watchObservedRunningTime="2026-07-27 20:36:10.10802965 +0000 UTC m=+63.988178356"
Jul 27 20:36:12.073943 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.073899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j" event={"ID":"f634b6f0-ffb0-4dc2-a34d-e972be0430b6","Type":"ContainerStarted","Data":"421673664a50d9c452866beed452659800a2fc8508334ed7bc5467b0f169c1f9"}
Jul 27 20:36:12.074393 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.074006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j" event={"ID":"f634b6f0-ffb0-4dc2-a34d-e972be0430b6","Type":"ContainerStarted","Data":"046f5142b44601fd914fe6d6376599a4e1b2acef9554bfa0141ce6dd7d126f97"}
Jul 27 20:36:12.074393 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.074044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j" event={"ID":"f634b6f0-ffb0-4dc2-a34d-e972be0430b6","Type":"ContainerStarted","Data":"0d026f55cae75fa211f16fcf091d127bd9dd592c51a1d2844415736e529aac3f"}
Jul 27 20:36:12.094589 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.094524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dc76f687f-fzp9j" podStartSLOduration=1.912415599 podStartE2EDuration="4.094510601s" totalImagesPullingTime="2.182095002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:12.093346719 +0000 UTC m=+65.973495423" watchObservedRunningTime="2026-07-27 20:36:12.094510601 +0000 UTC m=+65.974659306"
Jul 27 20:36:12.351656 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.351613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:36:12.351841 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.351749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:36:12.354197 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.354167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:36:12.354328 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.354167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:36:12.364264 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.364242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/421f2241-7f4c-446b-ae1a-0fd24b7046d7-original-pull-secret\") pod \"global-pull-secret-syncer-p4zdq\" (UID: \"421f2241-7f4c-446b-ae1a-0fd24b7046d7\") " pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:36:12.364465 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.364444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f35da8cb-24bc-4c87-9821-df562697ed0c-metrics-certs\") pod \"network-metrics-daemon-7qvwm\" (UID: \"f35da8cb-24bc-4c87-9821-df562697ed0c\") " pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:36:12.452789 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.452755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqrvq\""
Jul 27 20:36:12.452972 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.452867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:36:12.455069 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.455045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:36:12.460842 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.460819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qvwm"
Jul 27 20:36:12.460951 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.460868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p4zdq"
Jul 27 20:36:12.466052 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.466032    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:36:12.477115 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.477090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhg2\" (UniqueName: \"kubernetes.io/projected/1f39d1d7-d09a-4a99-ad3d-eeb603597991-kube-api-access-vjhg2\") pod \"network-check-target-htcjh\" (UID: \"1f39d1d7-d09a-4a99-ad3d-eeb603597991\") " pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:36:12.771513 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.771434    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pl2wx\""
Jul 27 20:36:12.780449 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.780416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:36:12.839924 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.839882    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c84fd48f7-7hq6h"]
Jul 27 20:36:12.867903 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.867869    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbf55fc5b-678bb"]
Jul 27 20:36:12.885486 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.885460    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbf55fc5b-678bb"]
Jul 27 20:36:12.885652 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.885600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957431 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-oauth-config\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2wr\" (UniqueName: \"kubernetes.io/projected/7db42668-3d95-4139-b850-8621183e9d7c-kube-api-access-kd2wr\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-service-ca\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-trusted-ca-bundle\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-oauth-serving-cert\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-console-config\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:12.957967 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:12.957718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-serving-cert\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.058954 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.058870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-oauth-config\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.058954 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.058938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2wr\" (UniqueName: \"kubernetes.io/projected/7db42668-3d95-4139-b850-8621183e9d7c-kube-api-access-kd2wr\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059196 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.058980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-service-ca\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059196 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-trusted-ca-bundle\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059196 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-oauth-serving-cert\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059196 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-console-config\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059196 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-serving-cert\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059914 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-service-ca\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.059914 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-oauth-serving-cert\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.060109 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.059951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-console-config\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.060109 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.060030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-trusted-ca-bundle\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.061685 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.061660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-oauth-config\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.061985 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.061966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-serving-cert\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.066518 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.066494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2wr\" (UniqueName: \"kubernetes.io/projected/7db42668-3d95-4139-b850-8621183e9d7c-kube-api-access-kd2wr\") pod \"console-6fbf55fc5b-678bb\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") " pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.198011 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.197975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:13.722985 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.722944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qvwm"]
Jul 27 20:36:13.858963 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:13.858924    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35da8cb_24bc_4c87_9821_df562697ed0c.slice/crio-46e9cd13de28b7529df662096219e76e261c74bab68b0ec0ecba6fa9532a4e74 WatchSource:0}: Error finding container 46e9cd13de28b7529df662096219e76e261c74bab68b0ec0ecba6fa9532a4e74: Status 404 returned error can't find the container with id 46e9cd13de28b7529df662096219e76e261c74bab68b0ec0ecba6fa9532a4e74
Jul 27 20:36:13.947516 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.947488    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p4zdq"]
Jul 27 20:36:13.949342 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.949318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-htcjh"]
Jul 27 20:36:13.950511 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:13.950493    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbf55fc5b-678bb"]
Jul 27 20:36:13.957889 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:13.957863    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421f2241_7f4c_446b_ae1a_0fd24b7046d7.slice/crio-d8ee25ec3665e34ceeca1940e06ba506665db1a29cde84a675c65a985ec36b40 WatchSource:0}: Error finding container d8ee25ec3665e34ceeca1940e06ba506665db1a29cde84a675c65a985ec36b40: Status 404 returned error can't find the container with id d8ee25ec3665e34ceeca1940e06ba506665db1a29cde84a675c65a985ec36b40
Jul 27 20:36:13.958174 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:13.958152    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f39d1d7_d09a_4a99_ad3d_eeb603597991.slice/crio-3606bf029561f347badc151d09b2342f51da8157608d9dde3d15b69ede6aaced WatchSource:0}: Error finding container 3606bf029561f347badc151d09b2342f51da8157608d9dde3d15b69ede6aaced: Status 404 returned error can't find the container with id 3606bf029561f347badc151d09b2342f51da8157608d9dde3d15b69ede6aaced
Jul 27 20:36:13.958501 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:36:13.958412    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db42668_3d95_4139_b850_8621183e9d7c.slice/crio-7034a8996a31d31f6f12a2e9ba1aeb17e32a5c0999eb7d1375b3343cd2a92236 WatchSource:0}: Error finding container 7034a8996a31d31f6f12a2e9ba1aeb17e32a5c0999eb7d1375b3343cd2a92236: Status 404 returned error can't find the container with id 7034a8996a31d31f6f12a2e9ba1aeb17e32a5c0999eb7d1375b3343cd2a92236
Jul 27 20:36:14.085842 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.085809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-htcjh" event={"ID":"1f39d1d7-d09a-4a99-ad3d-eeb603597991","Type":"ContainerStarted","Data":"3606bf029561f347badc151d09b2342f51da8157608d9dde3d15b69ede6aaced"}
Jul 27 20:36:14.086972 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.086924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p4zdq" event={"ID":"421f2241-7f4c-446b-ae1a-0fd24b7046d7","Type":"ContainerStarted","Data":"d8ee25ec3665e34ceeca1940e06ba506665db1a29cde84a675c65a985ec36b40"}
Jul 27 20:36:14.088993 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.088968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf55fc5b-678bb" event={"ID":"7db42668-3d95-4139-b850-8621183e9d7c","Type":"ContainerStarted","Data":"245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff"}
Jul 27 20:36:14.090305 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.089392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf55fc5b-678bb" event={"ID":"7db42668-3d95-4139-b850-8621183e9d7c","Type":"ContainerStarted","Data":"7034a8996a31d31f6f12a2e9ba1aeb17e32a5c0999eb7d1375b3343cd2a92236"}
Jul 27 20:36:14.091796 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.091776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"b52dba045b0f5edd9b2a888bf0ababfbaf30a1146f55f3ff7abfee3fc32cb85c"}
Jul 27 20:36:14.093075 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.093056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qvwm" event={"ID":"f35da8cb-24bc-4c87-9821-df562697ed0c","Type":"ContainerStarted","Data":"46e9cd13de28b7529df662096219e76e261c74bab68b0ec0ecba6fa9532a4e74"}
Jul 27 20:36:14.107113 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:14.107053    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fbf55fc5b-678bb" podStartSLOduration=2.107031905 podStartE2EDuration="2.107031905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:14.105243555 +0000 UTC m=+67.985392261" watchObservedRunningTime="2026-07-27 20:36:14.107031905 +0000 UTC m=+67.987180610"
Jul 27 20:36:15.102703 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:15.102604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"10cc94e1d7135105c1768ad6d53e52f9138a0c82dd56507a26a2213c4a42ac5d"}
Jul 27 20:36:16.108538 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:16.108500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qvwm" event={"ID":"f35da8cb-24bc-4c87-9821-df562697ed0c","Type":"ContainerStarted","Data":"e7c223df4223ed98564e5170c5631b1b799516762445cf4ba7d5e423a9f71aad"}
Jul 27 20:36:18.325981 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:18.325944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:20.126663 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.126618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qvwm" event={"ID":"f35da8cb-24bc-4c87-9821-df562697ed0c","Type":"ContainerStarted","Data":"7364d1ef044f11b8a0a9d1c9ebfa3d16f9994ddf0dc936ea48442fce53b255d3"}
Jul 27 20:36:20.128099 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.128072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-htcjh" event={"ID":"1f39d1d7-d09a-4a99-ad3d-eeb603597991","Type":"ContainerStarted","Data":"3d2f850928b34e8fe177347247494054a335ed3abdbd5a6619d9c938f4f9990b"}
Jul 27 20:36:20.128231 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.128179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:36:20.129404 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.129378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p4zdq" event={"ID":"421f2241-7f4c-446b-ae1a-0fd24b7046d7","Type":"ContainerStarted","Data":"3e863308277d1ba0de68e7d24f2110af718b443614320f0a65f5b919fbbf79cc"}
Jul 27 20:36:20.132644 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.132625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"2e9a65eff48c22b68ef9a99bbe42b4034b42a31ba644d78520e758326d4cb672"}
Jul 27 20:36:20.132741 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.132671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"a3cabd1f8a0aa75402852fe1652f3843424be4ca495f338eaac0065c858ecd80"}
Jul 27 20:36:20.132741 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.132697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"280f65496dd5d5a2534a57881e58eb1839c551c1a6b74a4c1f85e8e91bf5a5cf"}
Jul 27 20:36:20.132741 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.132722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51a9e039-0de1-42c6-bd59-3db18e32084a","Type":"ContainerStarted","Data":"1ecd60fe6e2c6327d532b4dd84a10c55e068b5e7750432f2dee1152d1d227300"}
Jul 27 20:36:20.142617 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.142578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7qvwm" podStartSLOduration=72.425511398 podStartE2EDuration="1m14.142542418s" totalImagesPullingTime="1.71703102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:20.141486462 +0000 UTC m=+74.021635181" watchObservedRunningTime="2026-07-27 20:36:20.142542418 +0000 UTC m=+74.022691122"
Jul 27 20:36:20.156703 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.156656    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-htcjh" podStartSLOduration=68.640569656 podStartE2EDuration="1m14.156643079s" totalImagesPullingTime="5.516073423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:20.155265338 +0000 UTC m=+74.035414044" watchObservedRunningTime="2026-07-27 20:36:20.156643079 +0000 UTC m=+74.036791784"
Jul 27 20:36:20.170809 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.170767    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p4zdq" podStartSLOduration=68.643999981 podStartE2EDuration="1m14.170753657s" totalImagesPullingTime="5.526753676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:20.169512658 +0000 UTC m=+74.049661363" watchObservedRunningTime="2026-07-27 20:36:20.170753657 +0000 UTC m=+74.050902367"
Jul 27 20:36:20.199185 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:20.199139    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.021752471 podStartE2EDuration="11.199121502s" totalImagesPullingTime="9.155980044s" totalInitContainerRuntime="21.388987ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:20.197694703 +0000 UTC m=+74.077843428" watchObservedRunningTime="2026-07-27 20:36:20.199121502 +0000 UTC m=+74.079270208"
Jul 27 20:36:23.198140 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:23.198109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:23.198631 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:23.198165    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:23.202762 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:23.202740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:24.153154 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:24.153131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:36:24.195900 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:24.195868    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7d96b6b9-bzjs4"]
Jul 27 20:36:24.733465 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:24.733434    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:36:26.067423 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.067358    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c88b4d96c-44wj6" podUID="f461c1eb-3ca6-438d-bf24-eddbcc7b1037" containerName="console" containerID="cri-o://24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3" gracePeriod=15
Jul 27 20:36:26.344146 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.344119    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:36:26.491029 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.490991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-serving-cert\") pod \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") "
Jul 27 20:36:26.491191 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-oauth-serving-cert\") pod \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") "
Jul 27 20:36:26.491191 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-service-ca\") pod \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") "
Jul 27 20:36:26.491191 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-oauth-config\") pod \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") "
Jul 27 20:36:26.491356 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-config\") pod \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") "
Jul 27 20:36:26.491356 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfwhc\" (UniqueName: \"kubernetes.io/projected/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-kube-api-access-pfwhc\") pod \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\" (UID: \"f461c1eb-3ca6-438d-bf24-eddbcc7b1037\") "
Jul 27 20:36:26.491505 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f461c1eb-3ca6-438d-bf24-eddbcc7b1037" (UID: "f461c1eb-3ca6-438d-bf24-eddbcc7b1037"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:26.491588 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-service-ca" (OuterVolumeSpecName: "service-ca") pod "f461c1eb-3ca6-438d-bf24-eddbcc7b1037" (UID: "f461c1eb-3ca6-438d-bf24-eddbcc7b1037"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:26.491588 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491569    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-config" (OuterVolumeSpecName: "console-config") pod "f461c1eb-3ca6-438d-bf24-eddbcc7b1037" (UID: "f461c1eb-3ca6-438d-bf24-eddbcc7b1037"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:26.491709 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491693    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:26.491744 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491716    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-oauth-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:26.491744 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.491733    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:26.493428 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.493406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f461c1eb-3ca6-438d-bf24-eddbcc7b1037" (UID: "f461c1eb-3ca6-438d-bf24-eddbcc7b1037"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:26.493512 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.493465    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-kube-api-access-pfwhc" (OuterVolumeSpecName: "kube-api-access-pfwhc") pod "f461c1eb-3ca6-438d-bf24-eddbcc7b1037" (UID: "f461c1eb-3ca6-438d-bf24-eddbcc7b1037"). InnerVolumeSpecName "kube-api-access-pfwhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:26.493512 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.493480    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f461c1eb-3ca6-438d-bf24-eddbcc7b1037" (UID: "f461c1eb-3ca6-438d-bf24-eddbcc7b1037"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:26.592154 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.592126    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-oauth-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:26.592154 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.592156    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwhc\" (UniqueName: \"kubernetes.io/projected/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-kube-api-access-pfwhc\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:26.592341 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:26.592171    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f461c1eb-3ca6-438d-bf24-eddbcc7b1037-console-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:27.164811 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.164771    2581 generic.go:356] "Generic (PLEG): container finished" podID="f461c1eb-3ca6-438d-bf24-eddbcc7b1037" containerID="24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3" exitCode=0
Jul 27 20:36:27.165309 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.164824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88b4d96c-44wj6" event={"ID":"f461c1eb-3ca6-438d-bf24-eddbcc7b1037","Type":"ContainerDied","Data":"24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3"}
Jul 27 20:36:27.165309 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.164838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c88b4d96c-44wj6"
Jul 27 20:36:27.165309 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.164895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c88b4d96c-44wj6" event={"ID":"f461c1eb-3ca6-438d-bf24-eddbcc7b1037","Type":"ContainerDied","Data":"13287ce3de6289697409fd0bc21c695b2c1e9d4a193aa3e3592c98681f310a5b"}
Jul 27 20:36:27.165309 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.164918    2581 scope.go:117] "RemoveContainer" containerID="24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3"
Jul 27 20:36:27.172897 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.172882    2581 scope.go:117] "RemoveContainer" containerID="24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3"
Jul 27 20:36:27.173175 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:27.173155    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3\": container with ID starting with 24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3 not found: ID does not exist" containerID="24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3"
Jul 27 20:36:27.173240 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.173185    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3"} err="failed to get container status \"24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3\": rpc error: code = NotFound desc = could not find container \"24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3\": container with ID starting with 24cd4a8f20a83260aabd1a8f393970abdb717fd3aa0da8b7f1ffbf48154ecbd3 not found: ID does not exist"
Jul 27 20:36:27.180018 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.179999    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c88b4d96c-44wj6"]
Jul 27 20:36:27.185203 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:27.185184    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c88b4d96c-44wj6"]
Jul 27 20:36:28.733254 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:28.733218    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f461c1eb-3ca6-438d-bf24-eddbcc7b1037" path="/var/lib/kubelet/pods/f461c1eb-3ca6-438d-bf24-eddbcc7b1037/volumes"
Jul 27 20:36:37.865669 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:37.865612    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c84fd48f7-7hq6h" podUID="77bd1a68-055a-4072-99e9-8b8237bf211d" containerName="console" containerID="cri-o://699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa" gracePeriod=15
Jul 27 20:36:38.121947 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.121892    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:38.197638 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-serving-cert\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.197638 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxmfs\" (UniqueName: \"kubernetes.io/projected/77bd1a68-055a-4072-99e9-8b8237bf211d-kube-api-access-sxmfs\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.197852 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197665    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-trusted-ca-bundle\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.197852 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197705    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-console-config\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.197852 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-oauth-serving-cert\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.197852 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197790    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-service-ca\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.197852 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.197825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-oauth-config\") pod \"77bd1a68-055a-4072-99e9-8b8237bf211d\" (UID: \"77bd1a68-055a-4072-99e9-8b8237bf211d\") "
Jul 27 20:36:38.198238 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.198208    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.198349 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.198245    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.198349 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.198335    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-console-config" (OuterVolumeSpecName: "console-config") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.198441 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.198340    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-service-ca" (OuterVolumeSpecName: "service-ca") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.199871 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.199848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bd1a68-055a-4072-99e9-8b8237bf211d-kube-api-access-sxmfs" (OuterVolumeSpecName: "kube-api-access-sxmfs") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "kube-api-access-sxmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:38.199978 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.199956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:38.200050 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.200033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "77bd1a68-055a-4072-99e9-8b8237bf211d" (UID: "77bd1a68-055a-4072-99e9-8b8237bf211d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:38.213512 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.213489    2581 generic.go:356] "Generic (PLEG): container finished" podID="77bd1a68-055a-4072-99e9-8b8237bf211d" containerID="699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa" exitCode=0
Jul 27 20:36:38.213622 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.213539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c84fd48f7-7hq6h" event={"ID":"77bd1a68-055a-4072-99e9-8b8237bf211d","Type":"ContainerDied","Data":"699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa"}
Jul 27 20:36:38.213681 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.213625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c84fd48f7-7hq6h" event={"ID":"77bd1a68-055a-4072-99e9-8b8237bf211d","Type":"ContainerDied","Data":"5104649fa2fb39838649bc92adb436145533d42e479a826235e003e0445cfeab"}
Jul 27 20:36:38.213681 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.213645    2581 scope.go:117] "RemoveContainer" containerID="699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa"
Jul 27 20:36:38.213681 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.213560    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c84fd48f7-7hq6h"
Jul 27 20:36:38.222765 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.222749    2581 scope.go:117] "RemoveContainer" containerID="699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa"
Jul 27 20:36:38.223026 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:38.223000    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa\": container with ID starting with 699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa not found: ID does not exist" containerID="699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa"
Jul 27 20:36:38.223097 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.223033    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa"} err="failed to get container status \"699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa\": rpc error: code = NotFound desc = could not find container \"699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa\": container with ID starting with 699ddd80675de0f8fe36ddf3cbbe17bd5f8767a5d741687ad874a7775974dbfa not found: ID does not exist"
Jul 27 20:36:38.233299 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.233280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c84fd48f7-7hq6h"]
Jul 27 20:36:38.236876 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.236858    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c84fd48f7-7hq6h"]
Jul 27 20:36:38.298366 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298342    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-oauth-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.298366 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298364    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.298509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298374    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-oauth-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.298509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298383    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77bd1a68-055a-4072-99e9-8b8237bf211d-console-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.298509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298392    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmfs\" (UniqueName: \"kubernetes.io/projected/77bd1a68-055a-4072-99e9-8b8237bf211d-kube-api-access-sxmfs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.298509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298401    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-trusted-ca-bundle\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.298509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.298409    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77bd1a68-055a-4072-99e9-8b8237bf211d-console-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.733308 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:38.733274    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bd1a68-055a-4072-99e9-8b8237bf211d" path="/var/lib/kubelet/pods/77bd1a68-055a-4072-99e9-8b8237bf211d/volumes"
Jul 27 20:36:49.214857 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.214801    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f7d96b6b9-bzjs4" podUID="ad70af87-bb93-45de-ba89-a743143368b9" containerName="console" containerID="cri-o://f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1" gracePeriod=15
Jul 27 20:36:49.493457 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.493381    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:36:49.571641 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-console-config\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.571641 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-trusted-ca-bundle\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.571893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-oauth-config\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.571893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571727    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-serving-cert\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.571893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-service-ca\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.571893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg49g\" (UniqueName: \"kubernetes.io/projected/ad70af87-bb93-45de-ba89-a743143368b9-kube-api-access-pg49g\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.571893 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.571860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-oauth-serving-cert\") pod \"ad70af87-bb93-45de-ba89-a743143368b9\" (UID: \"ad70af87-bb93-45de-ba89-a743143368b9\") "
Jul 27 20:36:49.572203 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.572172    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:49.572322 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.572199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-console-config" (OuterVolumeSpecName: "console-config") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:49.572322 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.572226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:49.572395 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.572320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:49.574163 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.574139    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad70af87-bb93-45de-ba89-a743143368b9-kube-api-access-pg49g" (OuterVolumeSpecName: "kube-api-access-pg49g") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "kube-api-access-pg49g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:49.574432 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.574413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:49.574509 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.574492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ad70af87-bb93-45de-ba89-a743143368b9" (UID: "ad70af87-bb93-45de-ba89-a743143368b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:49.672849 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672816    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg49g\" (UniqueName: \"kubernetes.io/projected/ad70af87-bb93-45de-ba89-a743143368b9-kube-api-access-pg49g\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:49.672849 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672844    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-oauth-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:49.672849 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672855    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-console-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:49.672849 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672862    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-trusted-ca-bundle\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:49.673105 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672871    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-oauth-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:49.673105 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672881    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad70af87-bb93-45de-ba89-a743143368b9-console-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:49.673105 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:49.672893    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70af87-bb93-45de-ba89-a743143368b9-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:36:50.263581 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.263529    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad70af87-bb93-45de-ba89-a743143368b9" containerID="f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1" exitCode=0
Jul 27 20:36:50.263980 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.263591    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7d96b6b9-bzjs4"
Jul 27 20:36:50.263980 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.263619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7d96b6b9-bzjs4" event={"ID":"ad70af87-bb93-45de-ba89-a743143368b9","Type":"ContainerDied","Data":"f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1"}
Jul 27 20:36:50.263980 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.263664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7d96b6b9-bzjs4" event={"ID":"ad70af87-bb93-45de-ba89-a743143368b9","Type":"ContainerDied","Data":"5e2d72c41a4f76097c823fb855f331369f0143f5c11a55eb86931be3fd7d21e3"}
Jul 27 20:36:50.263980 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.263685    2581 scope.go:117] "RemoveContainer" containerID="f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1"
Jul 27 20:36:50.272695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.272679    2581 scope.go:117] "RemoveContainer" containerID="f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1"
Jul 27 20:36:50.272947 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:36:50.272925    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1\": container with ID starting with f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1 not found: ID does not exist" containerID="f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1"
Jul 27 20:36:50.273005 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.272959    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1"} err="failed to get container status \"f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1\": rpc error: code = NotFound desc = could not find container \"f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1\": container with ID starting with f7f88a9f9b96188b30a701cc9c5efbc483c9e021c36385dc413817f28b9412a1 not found: ID does not exist"
Jul 27 20:36:50.283677 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.283656    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7d96b6b9-bzjs4"]
Jul 27 20:36:50.287703 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.287683    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f7d96b6b9-bzjs4"]
Jul 27 20:36:50.732924 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:50.732894    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad70af87-bb93-45de-ba89-a743143368b9" path="/var/lib/kubelet/pods/ad70af87-bb93-45de-ba89-a743143368b9/volumes"
Jul 27 20:36:51.140462 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:36:51.140437    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-htcjh"
Jul 27 20:37:09.729395 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:09.729355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:37:09.748612 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:09.748584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:37:10.352009 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:10.351981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:37:25.318151 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318120    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66bf8947d8-vw655"]
Jul 27 20:37:25.318524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318409    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f461c1eb-3ca6-438d-bf24-eddbcc7b1037" containerName="console"
Jul 27 20:37:25.318524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f461c1eb-3ca6-438d-bf24-eddbcc7b1037" containerName="console"
Jul 27 20:37:25.318524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318469    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bd1a68-055a-4072-99e9-8b8237bf211d" containerName="console"
Jul 27 20:37:25.318524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318474    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bd1a68-055a-4072-99e9-8b8237bf211d" containerName="console"
Jul 27 20:37:25.318524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318482    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad70af87-bb93-45de-ba89-a743143368b9" containerName="console"
Jul 27 20:37:25.318524 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad70af87-bb93-45de-ba89-a743143368b9" containerName="console"
Jul 27 20:37:25.318725 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318533    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f461c1eb-3ca6-438d-bf24-eddbcc7b1037" containerName="console"
Jul 27 20:37:25.318725 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318542    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bd1a68-055a-4072-99e9-8b8237bf211d" containerName="console"
Jul 27 20:37:25.318725 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.318561    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad70af87-bb93-45de-ba89-a743143368b9" containerName="console"
Jul 27 20:37:25.321199 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.321184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.332411 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.332388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bf8947d8-vw655"]
Jul 27 20:37:25.472511 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-oauth-config\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.472699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-serving-cert\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.472699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-oauth-serving-cert\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.472699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tch\" (UniqueName: \"kubernetes.io/projected/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-kube-api-access-74tch\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.472699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-trusted-ca-bundle\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.472699 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-service-ca\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.472865 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.472740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-config\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573717 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-oauth-serving-cert\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573717 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tch\" (UniqueName: \"kubernetes.io/projected/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-kube-api-access-74tch\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573717 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-trusted-ca-bundle\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573969 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-service-ca\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573969 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-config\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573969 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-oauth-config\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.573969 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.573835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-serving-cert\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.574670 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.574650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-oauth-serving-cert\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.574790 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.574726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-service-ca\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.574790 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.574726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-config\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.574869 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.574810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-trusted-ca-bundle\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.576616 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.576597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-oauth-config\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.576711 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.576677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-serving-cert\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.580741 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.580721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tch\" (UniqueName: \"kubernetes.io/projected/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-kube-api-access-74tch\") pod \"console-66bf8947d8-vw655\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") " pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.629818 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.629784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:25.780026 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:25.779806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bf8947d8-vw655"]
Jul 27 20:37:25.783912 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:37:25.783882    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcccaaa6_ff76_4571_b17e_34d6cb8f2e1a.slice/crio-837b7f38cf8415e01d03ba4fe8995904b72f9c6c2df452cb38b4d7a022828f98 WatchSource:0}: Error finding container 837b7f38cf8415e01d03ba4fe8995904b72f9c6c2df452cb38b4d7a022828f98: Status 404 returned error can't find the container with id 837b7f38cf8415e01d03ba4fe8995904b72f9c6c2df452cb38b4d7a022828f98
Jul 27 20:37:26.402998 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:26.402961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bf8947d8-vw655" event={"ID":"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a","Type":"ContainerStarted","Data":"a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8"}
Jul 27 20:37:26.403375 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:26.403007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bf8947d8-vw655" event={"ID":"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a","Type":"ContainerStarted","Data":"837b7f38cf8415e01d03ba4fe8995904b72f9c6c2df452cb38b4d7a022828f98"}
Jul 27 20:37:26.419356 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:26.419312    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66bf8947d8-vw655" podStartSLOduration=1.419279542 podStartE2EDuration="1.419279542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:26.418449108 +0000 UTC m=+140.298597829" watchObservedRunningTime="2026-07-27 20:37:26.419279542 +0000 UTC m=+140.299428246"
Jul 27 20:37:35.630712 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:35.630677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:35.631119 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:35.630740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:35.635764 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:35.635741    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:36.442542 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:36.442510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:37:36.486104 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:37:36.486061    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbf55fc5b-678bb"]
Jul 27 20:38:01.510048 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.509947    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fbf55fc5b-678bb" podUID="7db42668-3d95-4139-b850-8621183e9d7c" containerName="console" containerID="cri-o://245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff" gracePeriod=15
Jul 27 20:38:01.740838 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.740818    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:38:01.879269 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd2wr\" (UniqueName: \"kubernetes.io/projected/7db42668-3d95-4139-b850-8621183e9d7c-kube-api-access-kd2wr\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879269 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-console-config\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879510 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879301    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-serving-cert\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879510 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-service-ca\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879652 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-oauth-serving-cert\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879652 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-oauth-config\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879756 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-trusted-ca-bundle\") pod \"7db42668-3d95-4139-b850-8621183e9d7c\" (UID: \"7db42668-3d95-4139-b850-8621183e9d7c\") "
Jul 27 20:38:01.879810 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-console-config" (OuterVolumeSpecName: "console-config") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:38:01.879864 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-service-ca" (OuterVolumeSpecName: "service-ca") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:38:01.879965 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879943    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-console-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:01.879965 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879967    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:01.880312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.879973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:38:01.880312 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.880232    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:38:01.881489 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.881468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:38:01.881848 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.881826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db42668-3d95-4139-b850-8621183e9d7c-kube-api-access-kd2wr" (OuterVolumeSpecName: "kube-api-access-kd2wr") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "kube-api-access-kd2wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:38:01.881924 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.881893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7db42668-3d95-4139-b850-8621183e9d7c" (UID: "7db42668-3d95-4139-b850-8621183e9d7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:38:01.980813 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.980779    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-trusted-ca-bundle\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:01.980813 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.980805    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd2wr\" (UniqueName: \"kubernetes.io/projected/7db42668-3d95-4139-b850-8621183e9d7c-kube-api-access-kd2wr\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:01.980813 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.980816    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:01.981028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.980824    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7db42668-3d95-4139-b850-8621183e9d7c-oauth-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:01.981028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:01.980833    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7db42668-3d95-4139-b850-8621183e9d7c-console-oauth-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:38:02.536655 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.536618    2581 generic.go:356] "Generic (PLEG): container finished" podID="7db42668-3d95-4139-b850-8621183e9d7c" containerID="245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff" exitCode=0
Jul 27 20:38:02.537032 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.536670    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf55fc5b-678bb"
Jul 27 20:38:02.537032 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.536667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf55fc5b-678bb" event={"ID":"7db42668-3d95-4139-b850-8621183e9d7c","Type":"ContainerDied","Data":"245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff"}
Jul 27 20:38:02.537032 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.536802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf55fc5b-678bb" event={"ID":"7db42668-3d95-4139-b850-8621183e9d7c","Type":"ContainerDied","Data":"7034a8996a31d31f6f12a2e9ba1aeb17e32a5c0999eb7d1375b3343cd2a92236"}
Jul 27 20:38:02.537032 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.536831    2581 scope.go:117] "RemoveContainer" containerID="245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff"
Jul 27 20:38:02.545022 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.545007    2581 scope.go:117] "RemoveContainer" containerID="245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff"
Jul 27 20:38:02.545268 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:38:02.545248    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff\": container with ID starting with 245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff not found: ID does not exist" containerID="245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff"
Jul 27 20:38:02.545326 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.545280    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff"} err="failed to get container status \"245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff\": rpc error: code = NotFound desc = could not find container \"245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff\": container with ID starting with 245f0ad52f7e1bd6fe3f2ab48ac8d9bc8be9a2d287e779928383191dcbc233ff not found: ID does not exist"
Jul 27 20:38:02.556055 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.556034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbf55fc5b-678bb"]
Jul 27 20:38:02.559860 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.559837    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbf55fc5b-678bb"]
Jul 27 20:38:02.733533 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:38:02.733503    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db42668-3d95-4139-b850-8621183e9d7c" path="/var/lib/kubelet/pods/7db42668-3d95-4139-b850-8621183e9d7c/volumes"
Jul 27 20:39:03.484915 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.484881    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"]
Jul 27 20:39:03.485342 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.485165    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db42668-3d95-4139-b850-8621183e9d7c" containerName="console"
Jul 27 20:39:03.485342 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.485177    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db42668-3d95-4139-b850-8621183e9d7c" containerName="console"
Jul 27 20:39:03.485342 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.485232    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db42668-3d95-4139-b850-8621183e9d7c" containerName="console"
Jul 27 20:39:03.487896 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.487881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.490246 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.490223    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 20:39:03.490920 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.490903    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nwt6n\""
Jul 27 20:39:03.490992 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.490928    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 20:39:03.497260 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.497239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"]
Jul 27 20:39:03.578035 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.577999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddb9\" (UniqueName: \"kubernetes.io/projected/3dac3fc7-456a-4518-b02e-2933a5a2410b-kube-api-access-fddb9\") pod \"cert-manager-cainjector-559cdd8c87-m4tv2\" (UID: \"3dac3fc7-456a-4518-b02e-2933a5a2410b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.578035 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.578044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3dac3fc7-456a-4518-b02e-2933a5a2410b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m4tv2\" (UID: \"3dac3fc7-456a-4518-b02e-2933a5a2410b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.678397 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.678358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3dac3fc7-456a-4518-b02e-2933a5a2410b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m4tv2\" (UID: \"3dac3fc7-456a-4518-b02e-2933a5a2410b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.678583 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.678443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddb9\" (UniqueName: \"kubernetes.io/projected/3dac3fc7-456a-4518-b02e-2933a5a2410b-kube-api-access-fddb9\") pod \"cert-manager-cainjector-559cdd8c87-m4tv2\" (UID: \"3dac3fc7-456a-4518-b02e-2933a5a2410b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.691688 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.691655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3dac3fc7-456a-4518-b02e-2933a5a2410b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m4tv2\" (UID: \"3dac3fc7-456a-4518-b02e-2933a5a2410b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.691804 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.691719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddb9\" (UniqueName: \"kubernetes.io/projected/3dac3fc7-456a-4518-b02e-2933a5a2410b-kube-api-access-fddb9\") pod \"cert-manager-cainjector-559cdd8c87-m4tv2\" (UID: \"3dac3fc7-456a-4518-b02e-2933a5a2410b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.812154 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.812077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"
Jul 27 20:39:03.932205 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:03.932169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2"]
Jul 27 20:39:03.935104 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:39:03.935078    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dac3fc7_456a_4518_b02e_2933a5a2410b.slice/crio-080fb043e052346d67c44ecf8f455c8d6a1072b3c24b5c5ba1b74865f9743530 WatchSource:0}: Error finding container 080fb043e052346d67c44ecf8f455c8d6a1072b3c24b5c5ba1b74865f9743530: Status 404 returned error can't find the container with id 080fb043e052346d67c44ecf8f455c8d6a1072b3c24b5c5ba1b74865f9743530
Jul 27 20:39:04.761416 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:04.761375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2" event={"ID":"3dac3fc7-456a-4518-b02e-2933a5a2410b","Type":"ContainerStarted","Data":"080fb043e052346d67c44ecf8f455c8d6a1072b3c24b5c5ba1b74865f9743530"}
Jul 27 20:39:07.774183 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:07.774149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2" event={"ID":"3dac3fc7-456a-4518-b02e-2933a5a2410b","Type":"ContainerStarted","Data":"75e307070ba02a88dc7cf07e00a07f79735a9d1daf64aaba7ca4e7d26fcd2494"}
Jul 27 20:39:07.795972 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:07.795925    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m4tv2" podStartSLOduration=2.00862658 podStartE2EDuration="4.795910774s" totalImagesPullingTime="2.787284194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:39:07.79401826 +0000 UTC m=+241.674166963" watchObservedRunningTime="2026-07-27 20:39:07.795910774 +0000 UTC m=+241.676059477"
Jul 27 20:39:12.565998 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.565967    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fbfls"]
Jul 27 20:39:12.569210 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.569195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.571681 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.571664    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ftb5s\""
Jul 27 20:39:12.578147 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.578124    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fbfls"]
Jul 27 20:39:12.654921 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.654891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jr8c\" (UniqueName: \"kubernetes.io/projected/69894ec4-a7b1-44e4-92ea-805fba7b2c16-kube-api-access-7jr8c\") pod \"cert-manager-55b4d954d5-fbfls\" (UID: \"69894ec4-a7b1-44e4-92ea-805fba7b2c16\") " pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.655087 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.654944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69894ec4-a7b1-44e4-92ea-805fba7b2c16-bound-sa-token\") pod \"cert-manager-55b4d954d5-fbfls\" (UID: \"69894ec4-a7b1-44e4-92ea-805fba7b2c16\") " pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.755894 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.755868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jr8c\" (UniqueName: \"kubernetes.io/projected/69894ec4-a7b1-44e4-92ea-805fba7b2c16-kube-api-access-7jr8c\") pod \"cert-manager-55b4d954d5-fbfls\" (UID: \"69894ec4-a7b1-44e4-92ea-805fba7b2c16\") " pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.756040 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.755914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69894ec4-a7b1-44e4-92ea-805fba7b2c16-bound-sa-token\") pod \"cert-manager-55b4d954d5-fbfls\" (UID: \"69894ec4-a7b1-44e4-92ea-805fba7b2c16\") " pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.765625 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.765600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69894ec4-a7b1-44e4-92ea-805fba7b2c16-bound-sa-token\") pod \"cert-manager-55b4d954d5-fbfls\" (UID: \"69894ec4-a7b1-44e4-92ea-805fba7b2c16\") " pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.766091 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.766072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jr8c\" (UniqueName: \"kubernetes.io/projected/69894ec4-a7b1-44e4-92ea-805fba7b2c16-kube-api-access-7jr8c\") pod \"cert-manager-55b4d954d5-fbfls\" (UID: \"69894ec4-a7b1-44e4-92ea-805fba7b2c16\") " pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.878137 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.878112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fbfls"
Jul 27 20:39:12.999158 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:12.999028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fbfls"]
Jul 27 20:39:13.001741 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:39:13.001713    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69894ec4_a7b1_44e4_92ea_805fba7b2c16.slice/crio-8ea88a13a892f42156d0a1d8431939704d2169bdd3b504ee8a73f84c27de7c9f WatchSource:0}: Error finding container 8ea88a13a892f42156d0a1d8431939704d2169bdd3b504ee8a73f84c27de7c9f: Status 404 returned error can't find the container with id 8ea88a13a892f42156d0a1d8431939704d2169bdd3b504ee8a73f84c27de7c9f
Jul 27 20:39:13.799450 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:13.799419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fbfls" event={"ID":"69894ec4-a7b1-44e4-92ea-805fba7b2c16","Type":"ContainerStarted","Data":"c271847ac6f2b360f6619d007697f3912ce5ddf51256a31eea0a9cabbdbb72ac"}
Jul 27 20:39:13.799450 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:13.799457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fbfls" event={"ID":"69894ec4-a7b1-44e4-92ea-805fba7b2c16","Type":"ContainerStarted","Data":"8ea88a13a892f42156d0a1d8431939704d2169bdd3b504ee8a73f84c27de7c9f"}
Jul 27 20:39:13.817299 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:13.817255    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fbfls" podStartSLOduration=1.8172418609999998 podStartE2EDuration="1.817241861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:39:13.815999656 +0000 UTC m=+247.696148360" watchObservedRunningTime="2026-07-27 20:39:13.817241861 +0000 UTC m=+247.697390564"
Jul 27 20:39:40.494712 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.494638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"]
Jul 27 20:39:40.539670 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.539631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"]
Jul 27 20:39:40.539813 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.539725    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.542139 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.542119    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:39:40.544478 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.544450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 20:39:40.544478 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.544463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 20:39:40.544631 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.544489    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-57br9\""
Jul 27 20:39:40.544631 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.544489    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 20:39:40.544830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.544817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 20:39:40.581471 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.581447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdb67\" (UniqueName: \"kubernetes.io/projected/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-kube-api-access-sdb67\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.581583 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.581476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-cert\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.581625 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.581585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-metrics-cert\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.581625 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.581609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-manager-config\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.682850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.682815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-metrics-cert\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.682850 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.682855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-manager-config\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.683069 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.682904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdb67\" (UniqueName: \"kubernetes.io/projected/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-kube-api-access-sdb67\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.683069 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.682931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-cert\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.683660 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.683642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-manager-config\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.685339 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.685319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-cert\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.685444 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.685388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-metrics-cert\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.692232 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.692213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdb67\" (UniqueName: \"kubernetes.io/projected/a73bf4d5-efa5-40ea-bc94-a44f39b40cd9-kube-api-access-sdb67\") pod \"lws-controller-manager-67d8f5d499-zfvf5\" (UID: \"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9\") " pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.849722 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.849692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:40.996051 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:40.996024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"]
Jul 27 20:39:40.998819 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:39:40.998790    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73bf4d5_efa5_40ea_bc94_a44f39b40cd9.slice/crio-98f83b505d31157ea96af383489f828a14a7915c79e32193194efb0ec0291e01 WatchSource:0}: Error finding container 98f83b505d31157ea96af383489f828a14a7915c79e32193194efb0ec0291e01: Status 404 returned error can't find the container with id 98f83b505d31157ea96af383489f828a14a7915c79e32193194efb0ec0291e01
Jul 27 20:39:41.901120 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:41.901072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5" event={"ID":"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9","Type":"ContainerStarted","Data":"98f83b505d31157ea96af383489f828a14a7915c79e32193194efb0ec0291e01"}
Jul 27 20:39:43.911749 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:43.911713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5" event={"ID":"a73bf4d5-efa5-40ea-bc94-a44f39b40cd9","Type":"ContainerStarted","Data":"527b22073d91653c8fb5d9923c15b487347e1703a6f7f6aecd0f5200fc0dadf3"}
Jul 27 20:39:43.912127 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:43.911774    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:43.928324 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:43.928277    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5" podStartSLOduration=1.606992499 podStartE2EDuration="3.928260705s" totalImagesPullingTime="2.321268206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:39:43.92729666 +0000 UTC m=+277.807445369" watchObservedRunningTime="2026-07-27 20:39:43.928260705 +0000 UTC m=+277.808409410"
Jul 27 20:39:51.968005 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.967969    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"]
Jul 27 20:39:51.971511 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.971496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:51.975359 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.975338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 20:39:51.975510 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.975491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ln5pk\""
Jul 27 20:39:51.975590 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.975513    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 20:39:51.975947 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.975934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 20:39:51.981673 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.981658    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 20:39:51.998150 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:51.998123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"]
Jul 27 20:39:52.084064 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.084025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6df8a48-972e-46ce-8e53-53575b19d1a7-tmp\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.084237 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.084072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rzv\" (UniqueName: \"kubernetes.io/projected/d6df8a48-972e-46ce-8e53-53575b19d1a7-kube-api-access-s8rzv\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.084237 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.084136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6df8a48-972e-46ce-8e53-53575b19d1a7-manifests\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.084237 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.084184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6df8a48-972e-46ce-8e53-53575b19d1a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.084337 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.084259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6df8a48-972e-46ce-8e53-53575b19d1a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.184909 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.184878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6df8a48-972e-46ce-8e53-53575b19d1a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.185067 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.184928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6df8a48-972e-46ce-8e53-53575b19d1a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.185067 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.184957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6df8a48-972e-46ce-8e53-53575b19d1a7-tmp\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.185067 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.184983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rzv\" (UniqueName: \"kubernetes.io/projected/d6df8a48-972e-46ce-8e53-53575b19d1a7-kube-api-access-s8rzv\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.185067 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.185009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6df8a48-972e-46ce-8e53-53575b19d1a7-manifests\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.185417 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.185390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6df8a48-972e-46ce-8e53-53575b19d1a7-manifests\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.187329 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.187301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6df8a48-972e-46ce-8e53-53575b19d1a7-tmp\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.187444 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.187365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6df8a48-972e-46ce-8e53-53575b19d1a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.187444 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.187393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6df8a48-972e-46ce-8e53-53575b19d1a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.197278 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.197255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rzv\" (UniqueName: \"kubernetes.io/projected/d6df8a48-972e-46ce-8e53-53575b19d1a7-kube-api-access-s8rzv\") pod \"opendatahub-operator-controller-manager-9b945668b-j8zhj\" (UID: \"d6df8a48-972e-46ce-8e53-53575b19d1a7\") " pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.281478 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.281393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:52.418219 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.418178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"]
Jul 27 20:39:52.422840 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:39:52.422816    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6df8a48_972e_46ce_8e53_53575b19d1a7.slice/crio-b740b9e46e4f45a50f2182a4ff0158c2c0f26db34ed2bda954332178f7c5add8 WatchSource:0}: Error finding container b740b9e46e4f45a50f2182a4ff0158c2c0f26db34ed2bda954332178f7c5add8: Status 404 returned error can't find the container with id b740b9e46e4f45a50f2182a4ff0158c2c0f26db34ed2bda954332178f7c5add8
Jul 27 20:39:52.946694 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:52.946659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj" event={"ID":"d6df8a48-972e-46ce-8e53-53575b19d1a7","Type":"ContainerStarted","Data":"b740b9e46e4f45a50f2182a4ff0158c2c0f26db34ed2bda954332178f7c5add8"}
Jul 27 20:39:54.918242 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:54.918209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67d8f5d499-zfvf5"
Jul 27 20:39:55.964291 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:55.964261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj" event={"ID":"d6df8a48-972e-46ce-8e53-53575b19d1a7","Type":"ContainerStarted","Data":"f3e7073e417090a3a3a4ff993db7d9b0ebed27d49e6b9eabacb589af7c07fc1a"}
Jul 27 20:39:56.969574 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:56.969528    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6df8a48-972e-46ce-8e53-53575b19d1a7" containerID="f3e7073e417090a3a3a4ff993db7d9b0ebed27d49e6b9eabacb589af7c07fc1a" exitCode=0
Jul 27 20:39:56.969925 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:56.969611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj" event={"ID":"d6df8a48-972e-46ce-8e53-53575b19d1a7","Type":"ContainerDied","Data":"f3e7073e417090a3a3a4ff993db7d9b0ebed27d49e6b9eabacb589af7c07fc1a"}
Jul 27 20:39:57.976565 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:57.976508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj" event={"ID":"d6df8a48-972e-46ce-8e53-53575b19d1a7","Type":"ContainerStarted","Data":"9a630f0adbdffe015fe497e0a881fb5b025b91d87e8a9555d67556175c7bd52a"}
Jul 27 20:39:57.976973 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:57.976710    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:39:58.029141 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:39:58.029092    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj" podStartSLOduration=2.094216489 podStartE2EDuration="7.029075383s" totalImagesPullingTime="3.925399507s" totalInitContainerRuntime="1.009459387s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:39:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:39:58.028701877 +0000 UTC m=+291.908850580" watchObservedRunningTime="2026-07-27 20:39:58.029075383 +0000 UTC m=+291.909224087"
Jul 27 20:40:06.528225 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:06.528200    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:40:06.534931 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:06.534899    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:40:06.535739 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:06.535720    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 20:40:08.982746 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:08.982718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9b945668b-j8zhj"
Jul 27 20:40:18.547712 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.547679    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"]
Jul 27 20:40:18.550952 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.550930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.554542 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.554518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 20:40:18.554663 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.554603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:40:18.555362 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.555341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c4bjm\""
Jul 27 20:40:18.555490 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.555469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 20:40:18.555591 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.555575    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 20:40:18.588076 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.588048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"]
Jul 27 20:40:18.617439 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.617409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3230854-bed5-41e3-84ba-bd08d899837b-tmp\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.617611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.617459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgkd\" (UniqueName: \"kubernetes.io/projected/d3230854-bed5-41e3-84ba-bd08d899837b-kube-api-access-mzgkd\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.617611 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.617526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3230854-bed5-41e3-84ba-bd08d899837b-tls-certs\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.718153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.718115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3230854-bed5-41e3-84ba-bd08d899837b-tls-certs\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.718336 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.718183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3230854-bed5-41e3-84ba-bd08d899837b-tmp\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.718336 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.718217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkd\" (UniqueName: \"kubernetes.io/projected/d3230854-bed5-41e3-84ba-bd08d899837b-kube-api-access-mzgkd\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.718336 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:40:18.718264    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 20:40:18.718492 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:40:18.718344    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3230854-bed5-41e3-84ba-bd08d899837b-tls-certs podName:d3230854-bed5-41e3-84ba-bd08d899837b nodeName:}" failed. No retries permitted until 2026-07-27 20:40:19.218321032 +0000 UTC m=+313.098469731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d3230854-bed5-41e3-84ba-bd08d899837b-tls-certs") pod "kube-auth-proxy-86f57966b4-dw86t" (UID: "d3230854-bed5-41e3-84ba-bd08d899837b") : secret "kube-auth-proxy-tls" not found
Jul 27 20:40:18.720472 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.720445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3230854-bed5-41e3-84ba-bd08d899837b-tmp\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:18.729354 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:18.729332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgkd\" (UniqueName: \"kubernetes.io/projected/d3230854-bed5-41e3-84ba-bd08d899837b-kube-api-access-mzgkd\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:19.221572 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:19.221519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3230854-bed5-41e3-84ba-bd08d899837b-tls-certs\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:19.223890 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:19.223866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3230854-bed5-41e3-84ba-bd08d899837b-tls-certs\") pod \"kube-auth-proxy-86f57966b4-dw86t\" (UID: \"d3230854-bed5-41e3-84ba-bd08d899837b\") " pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:19.460720 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:19.460679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"
Jul 27 20:40:19.630228 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:19.630198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86f57966b4-dw86t"]
Jul 27 20:40:19.633179 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:40:19.633148    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3230854_bed5_41e3_84ba_bd08d899837b.slice/crio-979ca2472320a10ac17dbaca4c413a138dff1f8e50d82e15d6968adadb45c13e WatchSource:0}: Error finding container 979ca2472320a10ac17dbaca4c413a138dff1f8e50d82e15d6968adadb45c13e: Status 404 returned error can't find the container with id 979ca2472320a10ac17dbaca4c413a138dff1f8e50d82e15d6968adadb45c13e
Jul 27 20:40:19.634876 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:19.634856    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:40:20.069402 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:20.069363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t" event={"ID":"d3230854-bed5-41e3-84ba-bd08d899837b","Type":"ContainerStarted","Data":"979ca2472320a10ac17dbaca4c413a138dff1f8e50d82e15d6968adadb45c13e"}
Jul 27 20:40:23.083004 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:23.082962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t" event={"ID":"d3230854-bed5-41e3-84ba-bd08d899837b","Type":"ContainerStarted","Data":"269f1f624139f6f983fd576c683f7625814b739ae49fed5c8a80a4fb7a9e0a2b"}
Jul 27 20:40:23.146254 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:40:23.146201    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86f57966b4-dw86t" podStartSLOduration=2.499836031 podStartE2EDuration="5.146184729s" totalImagesPullingTime="2.646348698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:40:23.145394566 +0000 UTC m=+317.025543288" watchObservedRunningTime="2026-07-27 20:40:23.146184729 +0000 UTC m=+317.026333433"
Jul 27 20:41:14.054447 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.054362    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66dd8cb49f-gkvp6"]
Jul 27 20:41:14.057695 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.057676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.067663 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.067642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66dd8cb49f-gkvp6"]
Jul 27 20:41:14.086568 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-trusted-ca-bundle\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.086730 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqp62\" (UniqueName: \"kubernetes.io/projected/4680218e-d24c-4ae1-af4b-3434f079e22e-kube-api-access-qqp62\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.086730 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4680218e-d24c-4ae1-af4b-3434f079e22e-console-oauth-config\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.086730 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4680218e-d24c-4ae1-af4b-3434f079e22e-console-serving-cert\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.086925 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-service-ca\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.086925 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-console-config\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.086925 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.086793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-oauth-serving-cert\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.187673 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-service-ca\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.187673 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-console-config\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.187859 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-oauth-serving-cert\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.187859 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-trusted-ca-bundle\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.187859 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqp62\" (UniqueName: \"kubernetes.io/projected/4680218e-d24c-4ae1-af4b-3434f079e22e-kube-api-access-qqp62\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.187963 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4680218e-d24c-4ae1-af4b-3434f079e22e-console-oauth-config\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.188012 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.187969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4680218e-d24c-4ae1-af4b-3434f079e22e-console-serving-cert\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.188481 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.188458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-console-config\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.188644 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.188599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-oauth-serving-cert\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.188644 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.188599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-service-ca\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.188760 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.188651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4680218e-d24c-4ae1-af4b-3434f079e22e-trusted-ca-bundle\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.190436 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.190402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4680218e-d24c-4ae1-af4b-3434f079e22e-console-oauth-config\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.190436 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.190426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4680218e-d24c-4ae1-af4b-3434f079e22e-console-serving-cert\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.195555 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.195518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqp62\" (UniqueName: \"kubernetes.io/projected/4680218e-d24c-4ae1-af4b-3434f079e22e-kube-api-access-qqp62\") pod \"console-66dd8cb49f-gkvp6\" (UID: \"4680218e-d24c-4ae1-af4b-3434f079e22e\") " pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.365918 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.365887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:14.493381 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:14.493295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66dd8cb49f-gkvp6"]
Jul 27 20:41:14.496115 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:41:14.496088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4680218e_d24c_4ae1_af4b_3434f079e22e.slice/crio-987ba7848b2723d51d238bed329f894740da49d6725e9ee4d59aa4b35cf052c8 WatchSource:0}: Error finding container 987ba7848b2723d51d238bed329f894740da49d6725e9ee4d59aa4b35cf052c8: Status 404 returned error can't find the container with id 987ba7848b2723d51d238bed329f894740da49d6725e9ee4d59aa4b35cf052c8
Jul 27 20:41:15.309340 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:15.309304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dd8cb49f-gkvp6" event={"ID":"4680218e-d24c-4ae1-af4b-3434f079e22e","Type":"ContainerStarted","Data":"2b7bb1593289d98666d93649c7816b55c064f484ea7ed531695ff006a6c5c51b"}
Jul 27 20:41:15.309738 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:15.309353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66dd8cb49f-gkvp6" event={"ID":"4680218e-d24c-4ae1-af4b-3434f079e22e","Type":"ContainerStarted","Data":"987ba7848b2723d51d238bed329f894740da49d6725e9ee4d59aa4b35cf052c8"}
Jul 27 20:41:15.326236 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:15.326190    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66dd8cb49f-gkvp6" podStartSLOduration=1.326175121 podStartE2EDuration="1.326175121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:41:15.324496865 +0000 UTC m=+369.204645579" watchObservedRunningTime="2026-07-27 20:41:15.326175121 +0000 UTC m=+369.206323824"
Jul 27 20:41:24.366198 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:24.366153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:24.366689 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:24.366260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:24.370882 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:24.370863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:25.354083 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:25.354058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66dd8cb49f-gkvp6"
Jul 27 20:41:25.399108 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:25.399081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bf8947d8-vw655"]
Jul 27 20:41:50.421527 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.421470    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66bf8947d8-vw655" podUID="fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" containerName="console" containerID="cri-o://a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8" gracePeriod=15
Jul 27 20:41:50.654542 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.654519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:41:50.813862 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.813764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-serving-cert\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.813862 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.813821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-trusted-ca-bundle\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.814093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.813907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74tch\" (UniqueName: \"kubernetes.io/projected/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-kube-api-access-74tch\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.814093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.813951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-oauth-config\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.814093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.813977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-service-ca\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.814093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.814010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-oauth-serving-cert\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.814093 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.814030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-config\") pod \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\" (UID: \"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a\") "
Jul 27 20:41:50.814388 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.814361    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:41:50.814502 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.814471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-service-ca" (OuterVolumeSpecName: "service-ca") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:41:50.814655 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.814629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-config" (OuterVolumeSpecName: "console-config") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:41:50.814738 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.814705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:41:50.816067 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.816040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:41:50.816153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.816061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:41:50.816153 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.816132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-kube-api-access-74tch" (OuterVolumeSpecName: "kube-api-access-74tch") pod "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" (UID: "fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a"). InnerVolumeSpecName "kube-api-access-74tch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:41:50.915584 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915532    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:50.915584 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915577    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-trusted-ca-bundle\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:50.915584 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915586    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74tch\" (UniqueName: \"kubernetes.io/projected/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-kube-api-access-74tch\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:50.915584 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915595    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-oauth-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:50.915830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915604    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:50.915830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915612    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-oauth-serving-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:50.915830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:50.915622    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a-console-config\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:41:51.463014 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.462980    2581 generic.go:356] "Generic (PLEG): container finished" podID="fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" containerID="a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8" exitCode=0
Jul 27 20:41:51.463406 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.463021    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bf8947d8-vw655"
Jul 27 20:41:51.463406 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.463067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bf8947d8-vw655" event={"ID":"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a","Type":"ContainerDied","Data":"a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8"}
Jul 27 20:41:51.463406 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.463121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bf8947d8-vw655" event={"ID":"fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a","Type":"ContainerDied","Data":"837b7f38cf8415e01d03ba4fe8995904b72f9c6c2df452cb38b4d7a022828f98"}
Jul 27 20:41:51.463406 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.463141    2581 scope.go:117] "RemoveContainer" containerID="a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8"
Jul 27 20:41:51.471715 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.471690    2581 scope.go:117] "RemoveContainer" containerID="a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8"
Jul 27 20:41:51.471920 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:41:51.471904    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8\": container with ID starting with a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8 not found: ID does not exist" containerID="a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8"
Jul 27 20:41:51.471969 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.471927    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8"} err="failed to get container status \"a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8\": rpc error: code = NotFound desc = could not find container \"a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8\": container with ID starting with a68442e31b88eb90899977db5fbf5950b9dde834ad1e90a1826d8db641fbfec8 not found: ID does not exist"
Jul 27 20:41:51.482954 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.482933    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bf8947d8-vw655"]
Jul 27 20:41:51.494045 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:51.494015    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66bf8947d8-vw655"]
Jul 27 20:41:52.734007 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:52.733973    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" path="/var/lib/kubelet/pods/fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a/volumes"
Jul 27 20:41:57.098382 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.098350    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"]
Jul 27 20:41:57.098830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.098701    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" containerName="console"
Jul 27 20:41:57.098830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.098713    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" containerName="console"
Jul 27 20:41:57.098830 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.098770    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcccaaa6-ff76-4571-b17e-34d6cb8f2e1a" containerName="console"
Jul 27 20:41:57.103106 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.103088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:41:57.105618 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.105595    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 20:41:57.106302 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.106285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j4zx6\""
Jul 27 20:41:57.106356 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.106295    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 20:41:57.113623 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.113605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"]
Jul 27 20:41:57.156911 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.156878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pr4\" (UniqueName: \"kubernetes.io/projected/7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd-kube-api-access-29pr4\") pod \"authorino-operator-75cc8d5cb7-xlbvr\" (UID: \"7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:41:57.257755 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.257722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pr4\" (UniqueName: \"kubernetes.io/projected/7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd-kube-api-access-29pr4\") pod \"authorino-operator-75cc8d5cb7-xlbvr\" (UID: \"7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:41:57.268687 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.268655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pr4\" (UniqueName: \"kubernetes.io/projected/7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd-kube-api-access-29pr4\") pod \"authorino-operator-75cc8d5cb7-xlbvr\" (UID: \"7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:41:57.413644 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.413536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:41:57.540920 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:57.540893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"]
Jul 27 20:41:57.543632 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:41:57.543608    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcbe9d5_e8b5_4174_b8a9_3b1cdaffd2bd.slice/crio-c47123801692b84f228ec4d84e6a6bc89c792bc0029b3c53e3d2ac2552184ae3 WatchSource:0}: Error finding container c47123801692b84f228ec4d84e6a6bc89c792bc0029b3c53e3d2ac2552184ae3: Status 404 returned error can't find the container with id c47123801692b84f228ec4d84e6a6bc89c792bc0029b3c53e3d2ac2552184ae3
Jul 27 20:41:58.496628 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:41:58.496596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr" event={"ID":"7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd","Type":"ContainerStarted","Data":"c47123801692b84f228ec4d84e6a6bc89c792bc0029b3c53e3d2ac2552184ae3"}
Jul 27 20:42:00.506715 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:42:00.506677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr" event={"ID":"7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd","Type":"ContainerStarted","Data":"570f20184ae06e46fdc67cf50226b93494053c937ad19cf140bfa676b322f4e1"}
Jul 27 20:42:00.507096 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:42:00.506787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:42:00.523116 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:42:00.523075    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr" podStartSLOduration=1.3013189490000001 podStartE2EDuration="3.523062105s" totalImagesPullingTime="2.221743156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:42:00.522159456 +0000 UTC m=+414.402308174" watchObservedRunningTime="2026-07-27 20:42:00.523062105 +0000 UTC m=+414.403210809"
Jul 27 20:42:11.512675 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:42:11.512646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xlbvr"
Jul 27 20:43:08.525794 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.525761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:43:08.528969 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.528950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.531185 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.531163    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 20:43:08.531185 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.531180    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l2mk4\""
Jul 27 20:43:08.536986 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.536965    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:43:08.563457 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.563433    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:43:08.652427 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.652405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec2f3792-65ad-4461-acee-73be3fca1762-config-file\") pod \"limitador-limitador-54d6d988d5-4rf4s\" (UID: \"ec2f3792-65ad-4461-acee-73be3fca1762\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.652519 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.652465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrx6\" (UniqueName: \"kubernetes.io/projected/ec2f3792-65ad-4461-acee-73be3fca1762-kube-api-access-khrx6\") pod \"limitador-limitador-54d6d988d5-4rf4s\" (UID: \"ec2f3792-65ad-4461-acee-73be3fca1762\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.753266 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.753234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec2f3792-65ad-4461-acee-73be3fca1762-config-file\") pod \"limitador-limitador-54d6d988d5-4rf4s\" (UID: \"ec2f3792-65ad-4461-acee-73be3fca1762\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.753420 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.753384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrx6\" (UniqueName: \"kubernetes.io/projected/ec2f3792-65ad-4461-acee-73be3fca1762-kube-api-access-khrx6\") pod \"limitador-limitador-54d6d988d5-4rf4s\" (UID: \"ec2f3792-65ad-4461-acee-73be3fca1762\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.753922 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.753901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec2f3792-65ad-4461-acee-73be3fca1762-config-file\") pod \"limitador-limitador-54d6d988d5-4rf4s\" (UID: \"ec2f3792-65ad-4461-acee-73be3fca1762\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.760891 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.760863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrx6\" (UniqueName: \"kubernetes.io/projected/ec2f3792-65ad-4461-acee-73be3fca1762-kube-api-access-khrx6\") pod \"limitador-limitador-54d6d988d5-4rf4s\" (UID: \"ec2f3792-65ad-4461-acee-73be3fca1762\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.839603 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.839579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:08.965825 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:08.965704    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:43:09.795116 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:09.795073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s" event={"ID":"ec2f3792-65ad-4461-acee-73be3fca1762","Type":"ContainerStarted","Data":"5d853156e5024ebaac177ef7d92734a25a8ef89af057421bea601ee0b7a44e3a"}
Jul 27 20:43:11.805901 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:11.805816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s" event={"ID":"ec2f3792-65ad-4461-acee-73be3fca1762","Type":"ContainerStarted","Data":"78207ece6668a9a1e3f963dbaff6ee62bba146da5ea0cff11626c6a2f144f082"}
Jul 27 20:43:11.806258 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:11.805945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:43:11.822460 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:11.822417    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s" podStartSLOduration=1.388639843 podStartE2EDuration="3.822401182s" totalImagesPullingTime="2.433761339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:11.820742628 +0000 UTC m=+485.700891331" watchObservedRunningTime="2026-07-27 20:43:11.822401182 +0000 UTC m=+485.702549884"
Jul 27 20:43:22.811091 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:43:22.811062    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"
Jul 27 20:44:20.261367 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:44:20.261295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:44:41.739792 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:44:41.739758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:44:49.143393 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:44:49.143358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:44:56.939162 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:44:56.939129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:45:00.141890 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:00.141855    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"]
Jul 27 20:45:00.145347 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:00.145326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"
Jul 27 20:45:00.147595 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:00.147576    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 20:45:00.150442 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:00.150411    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"]
Jul 27 20:45:00.155372 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:00.155351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"
Jul 27 20:45:00.281478 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:00.281453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"]
Jul 27 20:45:00.283812 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:45:00.283784    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6b3b0b_ac95_43d6_83ea_4abd9a95aacc.slice/crio-2ee52464c8dc8372a4767abb5ad8c39d65940a71f15d3fdbf6cd08cd1853d8f6 WatchSource:0}: Error finding container 2ee52464c8dc8372a4767abb5ad8c39d65940a71f15d3fdbf6cd08cd1853d8f6: Status 404 returned error can't find the container with id 2ee52464c8dc8372a4767abb5ad8c39d65940a71f15d3fdbf6cd08cd1853d8f6
Jul 27 20:45:01.264519 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:01.264485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4" event={"ID":"6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc","Type":"ContainerStarted","Data":"2ee52464c8dc8372a4767abb5ad8c39d65940a71f15d3fdbf6cd08cd1853d8f6"}
Jul 27 20:45:03.277172 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:03.277077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4" event={"ID":"6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc","Type":"ContainerStarted","Data":"d273c711c1ae8b06c74976e0a68d4eaf503eee4443b23b064c8dba156cca836e"}
Jul 27 20:45:03.292325 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:03.292278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4" podStartSLOduration=0.972822729 podStartE2EDuration="3.29226333s" totalImagesPullingTime="2.319440601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:03.291767648 +0000 UTC m=+597.171916351" watchObservedRunningTime="2026-07-27 20:45:03.29226333 +0000 UTC m=+597.172412033"
Jul 27 20:45:04.282992 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:04.282957    2581 generic.go:356] "Generic (PLEG): container finished" podID="6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc" containerID="d273c711c1ae8b06c74976e0a68d4eaf503eee4443b23b064c8dba156cca836e" exitCode=0
Jul 27 20:45:04.283431 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:04.283049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4" event={"ID":"6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc","Type":"ContainerDied","Data":"d273c711c1ae8b06c74976e0a68d4eaf503eee4443b23b064c8dba156cca836e"}
Jul 27 20:45:05.413612 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:05.413587    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"
Jul 27 20:45:06.292841 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:06.292809    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4"
Jul 27 20:45:06.293024 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:06.292817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-wjcg4" event={"ID":"6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc","Type":"ContainerDied","Data":"2ee52464c8dc8372a4767abb5ad8c39d65940a71f15d3fdbf6cd08cd1853d8f6"}
Jul 27 20:45:06.293024 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:06.292935    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee52464c8dc8372a4767abb5ad8c39d65940a71f15d3fdbf6cd08cd1853d8f6"
Jul 27 20:45:06.558049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:06.557970    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:45:06.559857 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:06.559839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:45:17.839642 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:17.839608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:45:22.228411 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.228373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"]
Jul 27 20:45:22.228956 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.228935    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc" containerName="cleanup"
Jul 27 20:45:22.229028 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.228961    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc" containerName="cleanup"
Jul 27 20:45:22.229083 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.229068    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc" containerName="cleanup"
Jul 27 20:45:22.232434 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.232413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.235760 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.235734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 20:45:22.235883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.235767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 27 20:45:22.235883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.235777    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vz7np\""
Jul 27 20:45:22.235883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.235767    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 20:45:22.242975 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.242955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"]
Jul 27 20:45:22.381117 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.381083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.381296 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.381122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.381296 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.381147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f74ef3-04fb-418a-a25c-2120604d0cd7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.381296 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.381213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.381296 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.381263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.381296 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.381292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btml\" (UniqueName: \"kubernetes.io/projected/01f74ef3-04fb-418a-a25c-2120604d0cd7-kube-api-access-7btml\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482136 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482136 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482136 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f74ef3-04fb-418a-a25c-2120604d0cd7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482405 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482405 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482405 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btml\" (UniqueName: \"kubernetes.io/projected/01f74ef3-04fb-418a-a25c-2120604d0cd7-kube-api-access-7btml\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482542 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482705 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.482822 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.482786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.484902 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.484882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f74ef3-04fb-418a-a25c-2120604d0cd7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.485168 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.485151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f74ef3-04fb-418a-a25c-2120604d0cd7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.489570 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.489539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btml\" (UniqueName: \"kubernetes.io/projected/01f74ef3-04fb-418a-a25c-2120604d0cd7-kube-api-access-7btml\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-748hd\" (UID: \"01f74ef3-04fb-418a-a25c-2120604d0cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.542737 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.542704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:22.668412 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.668389    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"]
Jul 27 20:45:22.670838 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:45:22.670812    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f74ef3_04fb_418a_a25c_2120604d0cd7.slice/crio-4e3d0e65c24eefd9719cd30a7623b41b6012d69088e2a52895edcaded0629ba5 WatchSource:0}: Error finding container 4e3d0e65c24eefd9719cd30a7623b41b6012d69088e2a52895edcaded0629ba5: Status 404 returned error can't find the container with id 4e3d0e65c24eefd9719cd30a7623b41b6012d69088e2a52895edcaded0629ba5
Jul 27 20:45:22.672503 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:22.672483    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:45:23.371843 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:23.371806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd" event={"ID":"01f74ef3-04fb-418a-a25c-2120604d0cd7","Type":"ContainerStarted","Data":"4e3d0e65c24eefd9719cd30a7623b41b6012d69088e2a52895edcaded0629ba5"}
Jul 27 20:45:24.377078 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:24.377043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd" event={"ID":"01f74ef3-04fb-418a-a25c-2120604d0cd7","Type":"ContainerStarted","Data":"a70d259dc85bbf43a76d3375d33e64f91ac9a1d069eb04b0bb0869c21ff3e2af"}
Jul 27 20:45:24.395136 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:24.395089    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd" podStartSLOduration=1.262815416 podStartE2EDuration="2.39507389s" totalImagesPullingTime="1.132258474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:24.393170224 +0000 UTC m=+618.273318927" watchObservedRunningTime="2026-07-27 20:45:24.39507389 +0000 UTC m=+618.275222592"
Jul 27 20:45:25.043126 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:25.043090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:45:32.543851 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:32.543766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:32.543851 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:32.543837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:32.555863 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:32.555832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:33.426865 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:33.426837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-748hd"
Jul 27 20:45:40.931263 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:45:40.931226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:47:03.530348 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:47:03.530257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:47:13.738084 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:47:13.738049    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:47:24.436075 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:47:24.436033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:49:06.336078 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:49:06.336038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:49:19.333601 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:49:19.333563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:49:51.133430 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:49:51.133397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:00.946708 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:00.946628    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:06.586473 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:06.586446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:50:06.586905 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:06.586885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:50:11.238532 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:11.238494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:16.836713 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:16.836681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:28.236226 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:28.236189    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:37.338607 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:37.338572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:48.733427 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:48.733395    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:50:58.528373 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:50:58.528334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:51:20.639656 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:51:20.639624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:51:30.423732 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:51:30.423649    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:51:52.839520 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:51:52.839486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:52:06.135291 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:52:06.135252    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:52:22.138869 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:52:22.138829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:52:31.637133 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:52:31.637100    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:52:41.243801 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:52:41.243766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:52:50.636674 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:52:50.636642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:52:59.634957 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:52:59.634878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:54:50.130506 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:54:50.130468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:55:01.744332 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:01.744297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:55:06.616113 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:06.616083    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:55:06.617232 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:06.617214    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 20:55:11.033341 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:11.033307    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:55:22.740370 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:22.740335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:55:32.532915 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:32.532877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:55:43.931637 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:43.931602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:55:52.831890 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:55:52.831859    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:56:04.241004 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:56:04.240925    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:56:14.034450 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:56:14.034410    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:56:24.338665 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:56:24.338627    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:56:36.234366 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:56:36.234333    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:56:58.438162 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:56:58.438125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:57:09.790714 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.790682    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"]
Jul 27 20:57:09.793852 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.793831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:09.796831 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.796809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bf44ac6e-kserve-self-signed-certs\""
Jul 27 20:57:09.819339 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.819314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"]
Jul 27 20:57:09.901143 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.901116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-home\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:09.901250 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.901150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9gl\" (UniqueName: \"kubernetes.io/projected/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-kube-api-access-cv9gl\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:09.901250 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.901171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tmp-dir\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:09.901250 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.901243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-model-cache\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:09.901360 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.901297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-dshm\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:09.901360 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:09.901322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tls-certs\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.002679 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.002651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-dshm\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.002795 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.002686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tls-certs\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.002795 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.002732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-home\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.002795 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.002764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9gl\" (UniqueName: \"kubernetes.io/projected/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-kube-api-access-cv9gl\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.002952 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.002794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tmp-dir\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.002952 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.002844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-model-cache\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.003129 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.003101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-home\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.003189 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.003153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-model-cache\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.003189 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.003182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tmp-dir\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.004827 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.004811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-dshm\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.005072 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.005056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tls-certs\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.010923 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.010904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9gl\" (UniqueName: \"kubernetes.io/projected/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-kube-api-access-cv9gl\") pod \"e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") " pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.107566 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.107522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:10.233883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.233857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"]
Jul 27 20:57:10.235936 ip-10-0-142-163 kubenswrapper[2581]: W0727 20:57:10.235911    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod470d20db_b5a2_4ddc_8267_59fc76d3e0fd.slice/crio-107d16a417af816e21bf060a791cc13266be845a82e11fbf736aa44657c4f543 WatchSource:0}: Error finding container 107d16a417af816e21bf060a791cc13266be845a82e11fbf736aa44657c4f543: Status 404 returned error can't find the container with id 107d16a417af816e21bf060a791cc13266be845a82e11fbf736aa44657c4f543
Jul 27 20:57:10.412857 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.412783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp" event={"ID":"470d20db-b5a2-4ddc-8267-59fc76d3e0fd","Type":"ContainerStarted","Data":"fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3"}
Jul 27 20:57:10.412857 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.412821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp" event={"ID":"470d20db-b5a2-4ddc-8267-59fc76d3e0fd","Type":"ContainerStarted","Data":"107d16a417af816e21bf060a791cc13266be845a82e11fbf736aa44657c4f543"}
Jul 27 20:57:10.429922 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.429876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp" podStartSLOduration=1.429860724 podStartE2EDuration="1.429860724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:10.428695943 +0000 UTC m=+1324.308844646" watchObservedRunningTime="2026-07-27 20:57:10.429860724 +0000 UTC m=+1324.310009427"
Jul 27 20:57:10.632129 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:10.632096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:57:20.108442 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:20.108407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:20.108921 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:20.108451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:20.120465 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:20.120442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:20.465768 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:20.465695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:57:23.035097 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:23.035064    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:57:40.931962 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:57:40.931880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:58:01.855367 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:01.855335    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"]
Jul 27 20:58:04.734317 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:04.734287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:58:16.873773 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:16.873715    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp" podUID="470d20db-b5a2-4ddc-8267-59fc76d3e0fd" containerName="main" containerID="cri-o://fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3" gracePeriod=45
Jul 27 20:58:17.111913 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.111891    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:58:17.177883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.177801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-home\") pod \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") "
Jul 27 20:58:17.177883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.177837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv9gl\" (UniqueName: \"kubernetes.io/projected/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-kube-api-access-cv9gl\") pod \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") "
Jul 27 20:58:17.177883 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.177861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tls-certs\") pod \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") "
Jul 27 20:58:17.178132 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.177902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-dshm\") pod \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") "
Jul 27 20:58:17.178132 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.177968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tmp-dir\") pod \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") "
Jul 27 20:58:17.178132 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.177992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-model-cache\") pod \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\" (UID: \"470d20db-b5a2-4ddc-8267-59fc76d3e0fd\") "
Jul 27 20:58:17.178132 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.178110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-home" (OuterVolumeSpecName: "home") pod "470d20db-b5a2-4ddc-8267-59fc76d3e0fd" (UID: "470d20db-b5a2-4ddc-8267-59fc76d3e0fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:58:17.178317 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.178242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "470d20db-b5a2-4ddc-8267-59fc76d3e0fd" (UID: "470d20db-b5a2-4ddc-8267-59fc76d3e0fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:58:17.178317 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.178252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "470d20db-b5a2-4ddc-8267-59fc76d3e0fd" (UID: "470d20db-b5a2-4ddc-8267-59fc76d3e0fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:58:17.178317 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.178275    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:58:17.180124 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.180093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-dshm" (OuterVolumeSpecName: "dshm") pod "470d20db-b5a2-4ddc-8267-59fc76d3e0fd" (UID: "470d20db-b5a2-4ddc-8267-59fc76d3e0fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:58:17.180228 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.180128    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "470d20db-b5a2-4ddc-8267-59fc76d3e0fd" (UID: "470d20db-b5a2-4ddc-8267-59fc76d3e0fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:58:17.180228 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.180177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-kube-api-access-cv9gl" (OuterVolumeSpecName: "kube-api-access-cv9gl") pod "470d20db-b5a2-4ddc-8267-59fc76d3e0fd" (UID: "470d20db-b5a2-4ddc-8267-59fc76d3e0fd"). InnerVolumeSpecName "kube-api-access-cv9gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:58:17.279299 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.279270    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:58:17.279299 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.279294    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:58:17.279299 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.279305    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:58:17.279504 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.279313    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9gl\" (UniqueName: \"kubernetes.io/projected/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-kube-api-access-cv9gl\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:58:17.279504 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.279322    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470d20db-b5a2-4ddc-8267-59fc76d3e0fd-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 20:58:17.704484 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.704452    2581 generic.go:356] "Generic (PLEG): container finished" podID="470d20db-b5a2-4ddc-8267-59fc76d3e0fd" containerID="fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3" exitCode=0
Jul 27 20:58:17.704683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.704496    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"
Jul 27 20:58:17.704683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.704513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp" event={"ID":"470d20db-b5a2-4ddc-8267-59fc76d3e0fd","Type":"ContainerDied","Data":"fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3"}
Jul 27 20:58:17.704683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.704593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp" event={"ID":"470d20db-b5a2-4ddc-8267-59fc76d3e0fd","Type":"ContainerDied","Data":"107d16a417af816e21bf060a791cc13266be845a82e11fbf736aa44657c4f543"}
Jul 27 20:58:17.704683 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.704620    2581 scope.go:117] "RemoveContainer" containerID="fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3"
Jul 27 20:58:17.713049 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.713032    2581 scope.go:117] "RemoveContainer" containerID="fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3"
Jul 27 20:58:17.713296 ip-10-0-142-163 kubenswrapper[2581]: E0727 20:58:17.713278    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3\": container with ID starting with fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3 not found: ID does not exist" containerID="fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3"
Jul 27 20:58:17.713335 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.713309    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3"} err="failed to get container status \"fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3\": rpc error: code = NotFound desc = could not find container \"fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3\": container with ID starting with fe4ed30c6d17e31b5993d5310009d2408e6b2224620fe8f798cf556c0f39c4c3 not found: ID does not exist"
Jul 27 20:58:17.728920 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.728897    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"]
Jul 27 20:58:17.730674 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:17.730655    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bf44ac6e-kserve-f87769c59-88ntp"]
Jul 27 20:58:18.733239 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:18.733206    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="470d20db-b5a2-4ddc-8267-59fc76d3e0fd" path="/var/lib/kubelet/pods/470d20db-b5a2-4ddc-8267-59fc76d3e0fd/volumes"
Jul 27 20:58:19.842604 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:19.842570    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:58:27.432693 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:27.432658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:58:35.936003 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:35.935964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:58:43.529895 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:43.529860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:58:53.238361 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:58:53.238324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:59:02.733346 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:59:02.733275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:59:10.847040 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:59:10.847005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:59:20.740171 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:59:20.740141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:59:30.754641 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:59:30.754606    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:59:43.843985 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:59:43.843953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 20:59:53.345952 ip-10-0-142-163 kubenswrapper[2581]: I0727 20:59:53.345918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:00:00.135781 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.135732    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"]
Jul 27 21:00:00.136332 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.136236    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470d20db-b5a2-4ddc-8267-59fc76d3e0fd" containerName="main"
Jul 27 21:00:00.136332 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.136255    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="470d20db-b5a2-4ddc-8267-59fc76d3e0fd" containerName="main"
Jul 27 21:00:00.136451 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.136343    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="470d20db-b5a2-4ddc-8267-59fc76d3e0fd" containerName="main"
Jul 27 21:00:00.139435 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.139414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"
Jul 27 21:00:00.142075 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.142057    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:00:00.148694 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.148676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"
Jul 27 21:00:00.158853 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.158819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"]
Jul 27 21:00:00.281641 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:00.281611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"]
Jul 27 21:00:00.283841 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:00:00.283808    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0896817b_48bf_4a79_8824_43f86009fb5d.slice/crio-638f06d2ddaa406d5e19f4cf01ebc38db9e78964c6bfe8c58074bfc519644fb1 WatchSource:0}: Error finding container 638f06d2ddaa406d5e19f4cf01ebc38db9e78964c6bfe8c58074bfc519644fb1: Status 404 returned error can't find the container with id 638f06d2ddaa406d5e19f4cf01ebc38db9e78964c6bfe8c58074bfc519644fb1
Jul 27 21:00:01.150294 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:01.150264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc" event={"ID":"0896817b-48bf-4a79-8824-43f86009fb5d","Type":"ContainerStarted","Data":"a3aae8e05a474864dafbe740e6ecb768b42c5c35afd10a1d21b2a0e9416d11fa"}
Jul 27 21:00:01.150638 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:01.150321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc" event={"ID":"0896817b-48bf-4a79-8824-43f86009fb5d","Type":"ContainerStarted","Data":"638f06d2ddaa406d5e19f4cf01ebc38db9e78964c6bfe8c58074bfc519644fb1"}
Jul 27 21:00:01.177478 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:01.177433    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc" podStartSLOduration=1.177418169 podStartE2EDuration="1.177418169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:01.176990478 +0000 UTC m=+1495.057139182" watchObservedRunningTime="2026-07-27 21:00:01.177418169 +0000 UTC m=+1495.057566872"
Jul 27 21:00:02.156685 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:02.156647    2581 generic.go:356] "Generic (PLEG): container finished" podID="0896817b-48bf-4a79-8824-43f86009fb5d" containerID="a3aae8e05a474864dafbe740e6ecb768b42c5c35afd10a1d21b2a0e9416d11fa" exitCode=0
Jul 27 21:00:02.157052 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:02.156730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc" event={"ID":"0896817b-48bf-4a79-8824-43f86009fb5d","Type":"ContainerDied","Data":"a3aae8e05a474864dafbe740e6ecb768b42c5c35afd10a1d21b2a0e9416d11fa"}
Jul 27 21:00:03.284802 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:03.284776    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"
Jul 27 21:00:04.137410 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:04.137376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:00:04.166457 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:04.166432    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc"
Jul 27 21:00:04.166617 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:04.166426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-wflxc" event={"ID":"0896817b-48bf-4a79-8824-43f86009fb5d","Type":"ContainerDied","Data":"638f06d2ddaa406d5e19f4cf01ebc38db9e78964c6bfe8c58074bfc519644fb1"}
Jul 27 21:00:04.166617 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:04.166572    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638f06d2ddaa406d5e19f4cf01ebc38db9e78964c6bfe8c58074bfc519644fb1"
Jul 27 21:00:06.643607 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:06.643577    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:00:06.644928 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:06.644905    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:00:14.734318 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:14.734282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:00:25.433290 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:25.433256    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:00:32.040743 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:32.040670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:00:47.433710 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:47.433673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:00:59.137177 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:00:59.137141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:01:09.326588 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:01:09.326526    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:01:23.142450 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:01:23.142416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:01:32.440835 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:01:32.440800    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:01:42.750325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:01:42.750291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:01:55.838936 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:01:55.838905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:02:10.217040 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.216949    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"]
Jul 27 21:02:10.217484 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.217463    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0896817b-48bf-4a79-8824-43f86009fb5d" containerName="cleanup"
Jul 27 21:02:10.217574 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.217487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0896817b-48bf-4a79-8824-43f86009fb5d" containerName="cleanup"
Jul 27 21:02:10.217636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.217599    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0896817b-48bf-4a79-8824-43f86009fb5d" containerName="cleanup"
Jul 27 21:02:10.220509 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.220488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:10.223172 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.223148    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:02:10.223441 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.223416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:02:10.228048 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.228027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"]
Jul 27 21:02:10.301860 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.301817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cf25cc-9199-4af3-b743-b4e93de34461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q\" (UID: \"47cf25cc-9199-4af3-b743-b4e93de34461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:10.402743 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.402707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cf25cc-9199-4af3-b743-b4e93de34461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q\" (UID: \"47cf25cc-9199-4af3-b743-b4e93de34461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:10.403375 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.403355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cf25cc-9199-4af3-b743-b4e93de34461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q\" (UID: \"47cf25cc-9199-4af3-b743-b4e93de34461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:10.536675 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.536589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:10.673312 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.673285    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"]
Jul 27 21:02:10.675319 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:02:10.675294    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cf25cc_9199_4af3_b743_b4e93de34461.slice/crio-83edaeac8708cc257f5e28ce44117b61335b2fa12928d593a74c519fbc0e22b8 WatchSource:0}: Error finding container 83edaeac8708cc257f5e28ce44117b61335b2fa12928d593a74c519fbc0e22b8: Status 404 returned error can't find the container with id 83edaeac8708cc257f5e28ce44117b61335b2fa12928d593a74c519fbc0e22b8
Jul 27 21:02:10.734907 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:10.734876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q" event={"ID":"47cf25cc-9199-4af3-b743-b4e93de34461","Type":"ContainerStarted","Data":"83edaeac8708cc257f5e28ce44117b61335b2fa12928d593a74c519fbc0e22b8"}
Jul 27 21:02:11.740085 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:11.740008    2581 generic.go:356] "Generic (PLEG): container finished" podID="47cf25cc-9199-4af3-b743-b4e93de34461" containerID="d222342fa5ffb0d10af4ccd0c505017aae47fc7e442a0d70b0abccb2c4564b03" exitCode=0
Jul 27 21:02:11.740085 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:11.740052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q" event={"ID":"47cf25cc-9199-4af3-b743-b4e93de34461","Type":"ContainerDied","Data":"d222342fa5ffb0d10af4ccd0c505017aae47fc7e442a0d70b0abccb2c4564b03"}
Jul 27 21:02:12.869593 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:12.869568    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:12.919717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:12.919687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cf25cc-9199-4af3-b743-b4e93de34461-maas-api-service-ca\") pod \"47cf25cc-9199-4af3-b743-b4e93de34461\" (UID: \"47cf25cc-9199-4af3-b743-b4e93de34461\") "
Jul 27 21:02:12.920032 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:12.920011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cf25cc-9199-4af3-b743-b4e93de34461-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47cf25cc-9199-4af3-b743-b4e93de34461" (UID: "47cf25cc-9199-4af3-b743-b4e93de34461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:02:13.020671 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:13.020591    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cf25cc-9199-4af3-b743-b4e93de34461-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:02:13.750587 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:13.750556    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"
Jul 27 21:02:13.750587 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:13.750583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q" event={"ID":"47cf25cc-9199-4af3-b743-b4e93de34461","Type":"ContainerDied","Data":"83edaeac8708cc257f5e28ce44117b61335b2fa12928d593a74c519fbc0e22b8"}
Jul 27 21:02:13.751126 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:13.750639    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83edaeac8708cc257f5e28ce44117b61335b2fa12928d593a74c519fbc0e22b8"
Jul 27 21:02:24.034437 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:24.034398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:02:25.296898 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:25.296859    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"]
Jul 27 21:02:25.299079 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:25.299056    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dcdeec0f-bzk4q"]
Jul 27 21:02:26.733725 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:26.733693    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47cf25cc-9199-4af3-b743-b4e93de34461" path="/var/lib/kubelet/pods/47cf25cc-9199-4af3-b743-b4e93de34461/volumes"
Jul 27 21:02:34.927166 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:34.927135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:02:54.064783 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.064748    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"]
Jul 27 21:02:54.065181 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.065126    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47cf25cc-9199-4af3-b743-b4e93de34461" containerName="revoke-keys"
Jul 27 21:02:54.065181 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.065137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47cf25cc-9199-4af3-b743-b4e93de34461" containerName="revoke-keys"
Jul 27 21:02:54.065249 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.065199    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47cf25cc-9199-4af3-b743-b4e93de34461" containerName="revoke-keys"
Jul 27 21:02:54.068101 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.068086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:54.070397 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.070372    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:02:54.070501 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.070407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:02:54.075103 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.075073    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"]
Jul 27 21:02:54.180049 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.180013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c597238-d1e1-48ca-b445-d12010f30dd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6\" (UID: \"0c597238-d1e1-48ca-b445-d12010f30dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:54.280528 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.280492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c597238-d1e1-48ca-b445-d12010f30dd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6\" (UID: \"0c597238-d1e1-48ca-b445-d12010f30dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:54.281164 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.281146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c597238-d1e1-48ca-b445-d12010f30dd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6\" (UID: \"0c597238-d1e1-48ca-b445-d12010f30dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:54.379957 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.379928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:54.505944 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.505919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"]
Jul 27 21:02:54.508337 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:02:54.508309    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c597238_d1e1_48ca_b445_d12010f30dd5.slice/crio-27ec39c4311e25e858300e2e33f1693365d12939102de28ac51c974115ee94fe WatchSource:0}: Error finding container 27ec39c4311e25e858300e2e33f1693365d12939102de28ac51c974115ee94fe: Status 404 returned error can't find the container with id 27ec39c4311e25e858300e2e33f1693365d12939102de28ac51c974115ee94fe
Jul 27 21:02:54.934975 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.934889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6" event={"ID":"0c597238-d1e1-48ca-b445-d12010f30dd5","Type":"ContainerStarted","Data":"d38f306eccc2e0681262fdc29b8855b201a2628a7aa12d64b1d53617873c56be"}
Jul 27 21:02:54.934975 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.934940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6" event={"ID":"0c597238-d1e1-48ca-b445-d12010f30dd5","Type":"ContainerStarted","Data":"27ec39c4311e25e858300e2e33f1693365d12939102de28ac51c974115ee94fe"}
Jul 27 21:02:54.949657 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:54.949609    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6" podStartSLOduration=0.949587265 podStartE2EDuration="949.587265ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:54.94832245 +0000 UTC m=+1668.828471171" watchObservedRunningTime="2026-07-27 21:02:54.949587265 +0000 UTC m=+1668.829735972"
Jul 27 21:02:55.940494 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:55.940459    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c597238-d1e1-48ca-b445-d12010f30dd5" containerID="d38f306eccc2e0681262fdc29b8855b201a2628a7aa12d64b1d53617873c56be" exitCode=0
Jul 27 21:02:55.940865 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:55.940506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6" event={"ID":"0c597238-d1e1-48ca-b445-d12010f30dd5","Type":"ContainerDied","Data":"d38f306eccc2e0681262fdc29b8855b201a2628a7aa12d64b1d53617873c56be"}
Jul 27 21:02:57.081022 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.081002    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:57.206806 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.206719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c597238-d1e1-48ca-b445-d12010f30dd5-maas-api-service-ca\") pod \"0c597238-d1e1-48ca-b445-d12010f30dd5\" (UID: \"0c597238-d1e1-48ca-b445-d12010f30dd5\") "
Jul 27 21:02:57.207062 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.207040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c597238-d1e1-48ca-b445-d12010f30dd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c597238-d1e1-48ca-b445-d12010f30dd5" (UID: "0c597238-d1e1-48ca-b445-d12010f30dd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:02:57.307297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.307246    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c597238-d1e1-48ca-b445-d12010f30dd5-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:02:57.950023 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.949995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"
Jul 27 21:02:57.950199 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.950036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6" event={"ID":"0c597238-d1e1-48ca-b445-d12010f30dd5","Type":"ContainerDied","Data":"27ec39c4311e25e858300e2e33f1693365d12939102de28ac51c974115ee94fe"}
Jul 27 21:02:57.950199 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:02:57.950078    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ec39c4311e25e858300e2e33f1693365d12939102de28ac51c974115ee94fe"
Jul 27 21:03:07.538975 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:07.538937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:03:09.145322 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:09.145290    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"]
Jul 27 21:03:09.149703 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:09.149680    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d77cbc3-ljsb6"]
Jul 27 21:03:10.733973 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:10.733940    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c597238-d1e1-48ca-b445-d12010f30dd5" path="/var/lib/kubelet/pods/0c597238-d1e1-48ca-b445-d12010f30dd5/volumes"
Jul 27 21:03:18.734908 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:18.734878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:03:39.262305 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.262226    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"]
Jul 27 21:03:39.262809 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.262598    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c597238-d1e1-48ca-b445-d12010f30dd5" containerName="revoke-keys"
Jul 27 21:03:39.262809 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.262611    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c597238-d1e1-48ca-b445-d12010f30dd5" containerName="revoke-keys"
Jul 27 21:03:39.262809 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.262667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c597238-d1e1-48ca-b445-d12010f30dd5" containerName="revoke-keys"
Jul 27 21:03:39.265428 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.265413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:39.267841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.267817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:03:39.267841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.267819    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:03:39.272417 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.272391    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"]
Jul 27 21:03:39.382157 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.382124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ef4242-a032-485b-9116-4cac8771a3d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-df7343e0-8pswf\" (UID: \"12ef4242-a032-485b-9116-4cac8771a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:39.483278 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.483248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ef4242-a032-485b-9116-4cac8771a3d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-df7343e0-8pswf\" (UID: \"12ef4242-a032-485b-9116-4cac8771a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:39.483910 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.483889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ef4242-a032-485b-9116-4cac8771a3d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-df7343e0-8pswf\" (UID: \"12ef4242-a032-485b-9116-4cac8771a3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:39.576634 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.576527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:39.703973 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:39.703942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"]
Jul 27 21:03:39.707061 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:03:39.707034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ef4242_a032_485b_9116_4cac8771a3d0.slice/crio-80f7adfd0548452f9be7d6e9cb3aa66b884b47fa3305b6867c3a8ae1392a5ee6 WatchSource:0}: Error finding container 80f7adfd0548452f9be7d6e9cb3aa66b884b47fa3305b6867c3a8ae1392a5ee6: Status 404 returned error can't find the container with id 80f7adfd0548452f9be7d6e9cb3aa66b884b47fa3305b6867c3a8ae1392a5ee6
Jul 27 21:03:40.144791 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:40.144689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf" event={"ID":"12ef4242-a032-485b-9116-4cac8771a3d0","Type":"ContainerStarted","Data":"9932d905fe41ce1a88718d80d4f2801a6bad16b507106d0a6a6ac13adab12e1e"}
Jul 27 21:03:40.144791 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:40.144751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf" event={"ID":"12ef4242-a032-485b-9116-4cac8771a3d0","Type":"ContainerStarted","Data":"80f7adfd0548452f9be7d6e9cb3aa66b884b47fa3305b6867c3a8ae1392a5ee6"}
Jul 27 21:03:40.159364 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:40.159315    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf" podStartSLOduration=1.159299838 podStartE2EDuration="1.159299838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:40.158019681 +0000 UTC m=+1714.038168388" watchObservedRunningTime="2026-07-27 21:03:40.159299838 +0000 UTC m=+1714.039448543"
Jul 27 21:03:41.150690 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:41.150660    2581 generic.go:356] "Generic (PLEG): container finished" podID="12ef4242-a032-485b-9116-4cac8771a3d0" containerID="9932d905fe41ce1a88718d80d4f2801a6bad16b507106d0a6a6ac13adab12e1e" exitCode=0
Jul 27 21:03:41.151101 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:41.150723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf" event={"ID":"12ef4242-a032-485b-9116-4cac8771a3d0","Type":"ContainerDied","Data":"9932d905fe41ce1a88718d80d4f2801a6bad16b507106d0a6a6ac13adab12e1e"}
Jul 27 21:03:42.282520 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:42.282495    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:42.304539 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:42.304511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ef4242-a032-485b-9116-4cac8771a3d0-maas-api-service-ca\") pod \"12ef4242-a032-485b-9116-4cac8771a3d0\" (UID: \"12ef4242-a032-485b-9116-4cac8771a3d0\") "
Jul 27 21:03:42.304841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:42.304821    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ef4242-a032-485b-9116-4cac8771a3d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12ef4242-a032-485b-9116-4cac8771a3d0" (UID: "12ef4242-a032-485b-9116-4cac8771a3d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:03:42.405692 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:42.405603    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ef4242-a032-485b-9116-4cac8771a3d0-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:03:43.161187 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:43.161152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf" event={"ID":"12ef4242-a032-485b-9116-4cac8771a3d0","Type":"ContainerDied","Data":"80f7adfd0548452f9be7d6e9cb3aa66b884b47fa3305b6867c3a8ae1392a5ee6"}
Jul 27 21:03:43.161187 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:43.161177    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"
Jul 27 21:03:43.161389 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:43.161206    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80f7adfd0548452f9be7d6e9cb3aa66b884b47fa3305b6867c3a8ae1392a5ee6"
Jul 27 21:03:52.635230 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:52.635194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:03:54.338791 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:54.338760    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"]
Jul 27 21:03:54.343194 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:54.343168    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df7343e0-8pswf"]
Jul 27 21:03:54.734385 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:03:54.734311    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ef4242-a032-485b-9116-4cac8771a3d0" path="/var/lib/kubelet/pods/12ef4242-a032-485b-9116-4cac8771a3d0/volumes"
Jul 27 21:04:03.029828 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:03.029795    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:04:23.329521 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.329488    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"]
Jul 27 21:04:23.329920 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.329850    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ef4242-a032-485b-9116-4cac8771a3d0" containerName="revoke-keys"
Jul 27 21:04:23.329920 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.329863    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ef4242-a032-485b-9116-4cac8771a3d0" containerName="revoke-keys"
Jul 27 21:04:23.329995 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.329926    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ef4242-a032-485b-9116-4cac8771a3d0" containerName="revoke-keys"
Jul 27 21:04:23.332860 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.332844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:23.334946 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.334926    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:04:23.335076 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.335017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:04:23.338912 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.338890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"]
Jul 27 21:04:23.449405 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.449372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn\" (UID: \"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:23.550073 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.550043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn\" (UID: \"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:23.550729 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.550705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn\" (UID: \"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:23.644632 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.644601    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:23.768698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:23.768667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"]
Jul 27 21:04:23.772001 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:04:23.771978    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2b22d0_2f81_4e70_8bf1_58809ca3f1fa.slice/crio-11e63063fe0629863c44a2105561725a632ff2534590c50764c7930adbe64cb2 WatchSource:0}: Error finding container 11e63063fe0629863c44a2105561725a632ff2534590c50764c7930adbe64cb2: Status 404 returned error can't find the container with id 11e63063fe0629863c44a2105561725a632ff2534590c50764c7930adbe64cb2
Jul 27 21:04:24.359844 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:24.359811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn" event={"ID":"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa","Type":"ContainerStarted","Data":"92ac42c28b1d9f20872944e3702a0e7e5375c1ca960e1c3219ae090dcb728b19"}
Jul 27 21:04:24.360159 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:24.359868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn" event={"ID":"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa","Type":"ContainerStarted","Data":"11e63063fe0629863c44a2105561725a632ff2534590c50764c7930adbe64cb2"}
Jul 27 21:04:24.375480 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:24.375437    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn" podStartSLOduration=1.375421196 podStartE2EDuration="1.375421196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:24.373947451 +0000 UTC m=+1758.254096156" watchObservedRunningTime="2026-07-27 21:04:24.375421196 +0000 UTC m=+1758.255569903"
Jul 27 21:04:25.365914 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:25.365883    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa" containerID="92ac42c28b1d9f20872944e3702a0e7e5375c1ca960e1c3219ae090dcb728b19" exitCode=0
Jul 27 21:04:25.366262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:25.365964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn" event={"ID":"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa","Type":"ContainerDied","Data":"92ac42c28b1d9f20872944e3702a0e7e5375c1ca960e1c3219ae090dcb728b19"}
Jul 27 21:04:26.493290 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:26.493265    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:26.578260 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:26.578220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa-maas-api-service-ca\") pod \"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa\" (UID: \"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa\") "
Jul 27 21:04:26.578586 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:26.578542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa" (UID: "1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:04:26.679777 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:26.679695    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:04:27.375289 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:27.375259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"
Jul 27 21:04:27.375289 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:27.375284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn" event={"ID":"1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa","Type":"ContainerDied","Data":"11e63063fe0629863c44a2105561725a632ff2534590c50764c7930adbe64cb2"}
Jul 27 21:04:27.375493 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:27.375327    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11e63063fe0629863c44a2105561725a632ff2534590c50764c7930adbe64cb2"
Jul 27 21:04:37.128685 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:37.128646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:04:38.405226 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:38.405191    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"]
Jul 27 21:04:38.407798 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:38.407764    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-544e36ef-bcxbn"]
Jul 27 21:04:38.733665 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:04:38.733582    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa" path="/var/lib/kubelet/pods/1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa/volumes"
Jul 27 21:05:06.671351 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:05:06.671316    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:05:06.673640 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:05:06.673616    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:06:04.228353 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:04.228311    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:06:14.230288 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:14.230253    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:06:23.231338 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:23.231302    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:06:31.760774 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.760733    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"]
Jul 27 21:06:31.761242 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.761228    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa" containerName="revoke-keys"
Jul 27 21:06:31.761311 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.761248    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa" containerName="revoke-keys"
Jul 27 21:06:31.761360 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.761337    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2b22d0-2f81-4e70-8bf1-58809ca3f1fa" containerName="revoke-keys"
Jul 27 21:06:31.764329 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.764310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:31.769209 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.769187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:06:31.769742 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.769724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:06:31.791152 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.791126    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"]
Jul 27 21:06:31.868647 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.868607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d950c2-145a-415a-8c3a-dbd9bac49913-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt\" (UID: \"58d950c2-145a-415a-8c3a-dbd9bac49913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:31.969797 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.969766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d950c2-145a-415a-8c3a-dbd9bac49913-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt\" (UID: \"58d950c2-145a-415a-8c3a-dbd9bac49913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:31.970392 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:31.970370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d950c2-145a-415a-8c3a-dbd9bac49913-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt\" (UID: \"58d950c2-145a-415a-8c3a-dbd9bac49913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:32.074802 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:32.074697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:32.203335 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:32.203307    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"]
Jul 27 21:06:32.205055 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:06:32.205028    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d950c2_145a_415a_8c3a_dbd9bac49913.slice/crio-08724ab892699132b2dba7d70a9c54736cdb4aba27f2bf798289fe2ee6de01e2 WatchSource:0}: Error finding container 08724ab892699132b2dba7d70a9c54736cdb4aba27f2bf798289fe2ee6de01e2: Status 404 returned error can't find the container with id 08724ab892699132b2dba7d70a9c54736cdb4aba27f2bf798289fe2ee6de01e2
Jul 27 21:06:32.947510 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:32.947477    2581 generic.go:356] "Generic (PLEG): container finished" podID="58d950c2-145a-415a-8c3a-dbd9bac49913" containerID="6359eebfb2d2c0dd7a01bd0f1bf1686057562836c7e1b972c8a0717ea27ed8c4" exitCode=0
Jul 27 21:06:32.947852 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:32.947565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt" event={"ID":"58d950c2-145a-415a-8c3a-dbd9bac49913","Type":"ContainerDied","Data":"6359eebfb2d2c0dd7a01bd0f1bf1686057562836c7e1b972c8a0717ea27ed8c4"}
Jul 27 21:06:32.947852 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:32.947620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt" event={"ID":"58d950c2-145a-415a-8c3a-dbd9bac49913","Type":"ContainerStarted","Data":"08724ab892699132b2dba7d70a9c54736cdb4aba27f2bf798289fe2ee6de01e2"}
Jul 27 21:06:34.075893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.075872    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:34.192483 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.192440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d950c2-145a-415a-8c3a-dbd9bac49913-maas-api-service-ca\") pod \"58d950c2-145a-415a-8c3a-dbd9bac49913\" (UID: \"58d950c2-145a-415a-8c3a-dbd9bac49913\") "
Jul 27 21:06:34.192823 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.192800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d950c2-145a-415a-8c3a-dbd9bac49913-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58d950c2-145a-415a-8c3a-dbd9bac49913" (UID: "58d950c2-145a-415a-8c3a-dbd9bac49913"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:34.293504 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.293432    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d950c2-145a-415a-8c3a-dbd9bac49913-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:06:34.957302 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.957274    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"
Jul 27 21:06:34.957302 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.957289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt" event={"ID":"58d950c2-145a-415a-8c3a-dbd9bac49913","Type":"ContainerDied","Data":"08724ab892699132b2dba7d70a9c54736cdb4aba27f2bf798289fe2ee6de01e2"}
Jul 27 21:06:34.957496 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:34.957336    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08724ab892699132b2dba7d70a9c54736cdb4aba27f2bf798289fe2ee6de01e2"
Jul 27 21:06:44.829876 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:44.829840    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:06:46.795467 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:46.795429    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"]
Jul 27 21:06:46.798142 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:46.798116    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ded28f10-q8vvt"]
Jul 27 21:06:48.734038 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:06:48.734005    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d950c2-145a-415a-8c3a-dbd9bac49913" path="/var/lib/kubelet/pods/58d950c2-145a-415a-8c3a-dbd9bac49913/volumes"
Jul 27 21:07:22.530243 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:22.530200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:07:53.436700 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.436662    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"]
Jul 27 21:07:53.437187 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.437161    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d950c2-145a-415a-8c3a-dbd9bac49913" containerName="revoke-keys"
Jul 27 21:07:53.437187 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.437180    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d950c2-145a-415a-8c3a-dbd9bac49913" containerName="revoke-keys"
Jul 27 21:07:53.437277 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.437254    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d950c2-145a-415a-8c3a-dbd9bac49913" containerName="revoke-keys"
Jul 27 21:07:53.440212 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.440182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.443244 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.443221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9d99757d\"/\"default-dockercfg-vjktv\""
Jul 27 21:07:53.443363 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.443258    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9d99757d\"/\"e2e-lifecycle-model-9d99757d-kserve-self-signed-certs\""
Jul 27 21:07:53.443363 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.443282    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9d99757d\"/\"openshift-service-ca.crt\""
Jul 27 21:07:53.443363 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.443335    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9d99757d\"/\"kube-root-ca.crt\""
Jul 27 21:07:53.455349 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.455324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"]
Jul 27 21:07:53.488659 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.488626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-dshm\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.488817 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.488689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-home\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.488817 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.488715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tmp-dir\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.488817 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.488757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-model-cache\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.488918 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.488807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tls-certs\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.488918 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.488896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq59x\" (UniqueName: \"kubernetes.io/projected/88cf1bbb-7228-43ae-87fe-5446b27a7b06-kube-api-access-wq59x\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589572 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-model-cache\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589572 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tls-certs\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589784 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq59x\" (UniqueName: \"kubernetes.io/projected/88cf1bbb-7228-43ae-87fe-5446b27a7b06-kube-api-access-wq59x\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589784 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-dshm\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589784 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-home\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589784 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tmp-dir\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.589981 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.589960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-model-cache\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.590058 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.590040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-home\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.590108 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.590094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tmp-dir\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.591976 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.591951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-dshm\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.592130 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.592115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tls-certs\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.596868 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.596838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq59x\" (UniqueName: \"kubernetes.io/projected/88cf1bbb-7228-43ae-87fe-5446b27a7b06-kube-api-access-wq59x\") pod \"e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") " pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.751214 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.751110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:07:53.877805 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:53.877774    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"]
Jul 27 21:07:53.881442 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:07:53.881414    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88cf1bbb_7228_43ae_87fe_5446b27a7b06.slice/crio-17592b82d9e0262fa4cdae902acc24a0dda3f445054490bbbca2cedb3b4e8294 WatchSource:0}: Error finding container 17592b82d9e0262fa4cdae902acc24a0dda3f445054490bbbca2cedb3b4e8294: Status 404 returned error can't find the container with id 17592b82d9e0262fa4cdae902acc24a0dda3f445054490bbbca2cedb3b4e8294
Jul 27 21:07:54.324505 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:54.324472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58" event={"ID":"88cf1bbb-7228-43ae-87fe-5446b27a7b06","Type":"ContainerStarted","Data":"230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12"}
Jul 27 21:07:54.324691 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:54.324512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58" event={"ID":"88cf1bbb-7228-43ae-87fe-5446b27a7b06","Type":"ContainerStarted","Data":"17592b82d9e0262fa4cdae902acc24a0dda3f445054490bbbca2cedb3b4e8294"}
Jul 27 21:07:54.343469 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:54.343423    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58" podStartSLOduration=1.343408449 podStartE2EDuration="1.343408449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:54.341241791 +0000 UTC m=+1968.221390507" watchObservedRunningTime="2026-07-27 21:07:54.343408449 +0000 UTC m=+1968.223557154"
Jul 27 21:07:56.635743 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:07:56.635706    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:08:03.751979 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:03.751887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:08:03.751979 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:03.751941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:08:03.763938 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:03.763909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:08:04.380747 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:04.380718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:08:21.365212 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:21.365178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:08:25.638688 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.638656    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"]
Jul 27 21:08:25.642049 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.642032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:25.644265 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.644241    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:08:25.644367 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.644305    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:08:25.648324 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.648300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"]
Jul 27 21:08:25.676593 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.676535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cccfde08-685b-48b1-9035-2ab3e20039e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9d99757d-bdssg\" (UID: \"cccfde08-685b-48b1-9035-2ab3e20039e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:25.778067 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.778026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cccfde08-685b-48b1-9035-2ab3e20039e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9d99757d-bdssg\" (UID: \"cccfde08-685b-48b1-9035-2ab3e20039e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:25.778689 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.778669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cccfde08-685b-48b1-9035-2ab3e20039e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9d99757d-bdssg\" (UID: \"cccfde08-685b-48b1-9035-2ab3e20039e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:25.953306 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:25.953226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:26.076593 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:26.076566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"]
Jul 27 21:08:26.078689 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:08:26.078663    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccfde08_685b_48b1_9035_2ab3e20039e3.slice/crio-2763a9104399b83f6e30664929b68d7412e7d4f6121919bcf86633b9e1ca44ef WatchSource:0}: Error finding container 2763a9104399b83f6e30664929b68d7412e7d4f6121919bcf86633b9e1ca44ef: Status 404 returned error can't find the container with id 2763a9104399b83f6e30664929b68d7412e7d4f6121919bcf86633b9e1ca44ef
Jul 27 21:08:26.475405 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:26.475370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg" event={"ID":"cccfde08-685b-48b1-9035-2ab3e20039e3","Type":"ContainerStarted","Data":"eb1ece6cbfd1a521078b4d52c94c1d08bc95aca3092954af09adaef3f23ae7a6"}
Jul 27 21:08:26.475586 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:26.475427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg" event={"ID":"cccfde08-685b-48b1-9035-2ab3e20039e3","Type":"ContainerStarted","Data":"2763a9104399b83f6e30664929b68d7412e7d4f6121919bcf86633b9e1ca44ef"}
Jul 27 21:08:26.490040 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:26.489981    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg" podStartSLOduration=1.489968191 podStartE2EDuration="1.489968191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:08:26.48950985 +0000 UTC m=+2000.369658553" watchObservedRunningTime="2026-07-27 21:08:26.489968191 +0000 UTC m=+2000.370116894"
Jul 27 21:08:27.481838 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:27.481746    2581 generic.go:356] "Generic (PLEG): container finished" podID="cccfde08-685b-48b1-9035-2ab3e20039e3" containerID="eb1ece6cbfd1a521078b4d52c94c1d08bc95aca3092954af09adaef3f23ae7a6" exitCode=0
Jul 27 21:08:27.482220 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:27.481837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg" event={"ID":"cccfde08-685b-48b1-9035-2ab3e20039e3","Type":"ContainerDied","Data":"eb1ece6cbfd1a521078b4d52c94c1d08bc95aca3092954af09adaef3f23ae7a6"}
Jul 27 21:08:28.605242 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:28.605219    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:28.702297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:28.702268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cccfde08-685b-48b1-9035-2ab3e20039e3-maas-api-service-ca\") pod \"cccfde08-685b-48b1-9035-2ab3e20039e3\" (UID: \"cccfde08-685b-48b1-9035-2ab3e20039e3\") "
Jul 27 21:08:28.702653 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:28.702629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccfde08-685b-48b1-9035-2ab3e20039e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cccfde08-685b-48b1-9035-2ab3e20039e3" (UID: "cccfde08-685b-48b1-9035-2ab3e20039e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:08:28.803910 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:28.803875    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cccfde08-685b-48b1-9035-2ab3e20039e3-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:08:29.492598 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:29.492570    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"
Jul 27 21:08:29.492775 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:29.492605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg" event={"ID":"cccfde08-685b-48b1-9035-2ab3e20039e3","Type":"ContainerDied","Data":"2763a9104399b83f6e30664929b68d7412e7d4f6121919bcf86633b9e1ca44ef"}
Jul 27 21:08:29.492775 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:29.492662    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2763a9104399b83f6e30664929b68d7412e7d4f6121919bcf86633b9e1ca44ef"
Jul 27 21:08:39.434439 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:39.434407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:08:45.738260 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:45.738226    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"]
Jul 27 21:08:45.740560 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:45.740526    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d99757d-bdssg"]
Jul 27 21:08:46.738485 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:46.738451    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cccfde08-685b-48b1-9035-2ab3e20039e3" path="/var/lib/kubelet/pods/cccfde08-685b-48b1-9035-2ab3e20039e3/volumes"
Jul 27 21:08:48.229384 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:48.229349    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:08:53.892605 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:08:53.892573    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"]
Jul 27 21:09:06.744208 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:06.744180    2581 scope.go:117] "RemoveContainer" containerID="d38f306eccc2e0681262fdc29b8855b201a2628a7aa12d64b1d53617873c56be"
Jul 27 21:09:06.752387 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:06.752368    2581 scope.go:117] "RemoveContainer" containerID="d222342fa5ffb0d10af4ccd0c505017aae47fc7e442a0d70b0abccb2c4564b03"
Jul 27 21:09:08.910988 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:08.910924    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58" podUID="88cf1bbb-7228-43ae-87fe-5446b27a7b06" containerName="main" containerID="cri-o://230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12" gracePeriod=45
Jul 27 21:09:09.147795 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.147776    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:09:09.154351 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tls-certs\") pod \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") "
Jul 27 21:09:09.154428 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-dshm\") pod \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") "
Jul 27 21:09:09.154428 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tmp-dir\") pod \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") "
Jul 27 21:09:09.154428 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-home\") pod \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") "
Jul 27 21:09:09.154533 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-model-cache\") pod \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") "
Jul 27 21:09:09.154533 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154451    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq59x\" (UniqueName: \"kubernetes.io/projected/88cf1bbb-7228-43ae-87fe-5446b27a7b06-kube-api-access-wq59x\") pod \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\" (UID: \"88cf1bbb-7228-43ae-87fe-5446b27a7b06\") "
Jul 27 21:09:09.154716 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88cf1bbb-7228-43ae-87fe-5446b27a7b06" (UID: "88cf1bbb-7228-43ae-87fe-5446b27a7b06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:09.154716 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-home" (OuterVolumeSpecName: "home") pod "88cf1bbb-7228-43ae-87fe-5446b27a7b06" (UID: "88cf1bbb-7228-43ae-87fe-5446b27a7b06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:09.154835 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.154718    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-model-cache" (OuterVolumeSpecName: "model-cache") pod "88cf1bbb-7228-43ae-87fe-5446b27a7b06" (UID: "88cf1bbb-7228-43ae-87fe-5446b27a7b06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:09.156507 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.156487    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cf1bbb-7228-43ae-87fe-5446b27a7b06-kube-api-access-wq59x" (OuterVolumeSpecName: "kube-api-access-wq59x") pod "88cf1bbb-7228-43ae-87fe-5446b27a7b06" (UID: "88cf1bbb-7228-43ae-87fe-5446b27a7b06"). InnerVolumeSpecName "kube-api-access-wq59x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:09:09.156595 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.156542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-dshm" (OuterVolumeSpecName: "dshm") pod "88cf1bbb-7228-43ae-87fe-5446b27a7b06" (UID: "88cf1bbb-7228-43ae-87fe-5446b27a7b06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:09.156595 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.156562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88cf1bbb-7228-43ae-87fe-5446b27a7b06" (UID: "88cf1bbb-7228-43ae-87fe-5446b27a7b06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:09:09.256093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.256006    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:09:09.256093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.256036    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:09:09.256093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.256045    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:09:09.256093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.256051    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:09:09.256093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.256059    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cf1bbb-7228-43ae-87fe-5446b27a7b06-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:09:09.256093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.256068    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq59x\" (UniqueName: \"kubernetes.io/projected/88cf1bbb-7228-43ae-87fe-5446b27a7b06-kube-api-access-wq59x\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:09:09.680675 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.680640    2581 generic.go:356] "Generic (PLEG): container finished" podID="88cf1bbb-7228-43ae-87fe-5446b27a7b06" containerID="230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12" exitCode=0
Jul 27 21:09:09.680861 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.680681    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"
Jul 27 21:09:09.680861 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.680689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58" event={"ID":"88cf1bbb-7228-43ae-87fe-5446b27a7b06","Type":"ContainerDied","Data":"230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12"}
Jul 27 21:09:09.680861 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.680728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58" event={"ID":"88cf1bbb-7228-43ae-87fe-5446b27a7b06","Type":"ContainerDied","Data":"17592b82d9e0262fa4cdae902acc24a0dda3f445054490bbbca2cedb3b4e8294"}
Jul 27 21:09:09.680861 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.680748    2581 scope.go:117] "RemoveContainer" containerID="230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12"
Jul 27 21:09:09.689430 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.689411    2581 scope.go:117] "RemoveContainer" containerID="230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12"
Jul 27 21:09:09.689724 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:09:09.689705    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12\": container with ID starting with 230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12 not found: ID does not exist" containerID="230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12"
Jul 27 21:09:09.689783 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.689731    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12"} err="failed to get container status \"230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12\": rpc error: code = NotFound desc = could not find container \"230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12\": container with ID starting with 230e56b2abcedaf76dd66ceb982ea314dc2e7cdad919e2a76259ceb8f5d7ab12 not found: ID does not exist"
Jul 27 21:09:09.700580 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.700539    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"]
Jul 27 21:09:09.704401 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:09.704378    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9d99757d/e2e-lifecycle-model-9d99757d-kserve-5c48d6f979-cfg58"]
Jul 27 21:09:10.733528 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:10.733495    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88cf1bbb-7228-43ae-87fe-5446b27a7b06" path="/var/lib/kubelet/pods/88cf1bbb-7228-43ae-87fe-5446b27a7b06/volumes"
Jul 27 21:09:40.941033 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:40.940997    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:09:53.942852 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:09:53.942818    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:10:06.699896 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:10:06.699869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:10:06.702586 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:10:06.702567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:10:06.773532 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:10:06.773501    2581 scope.go:117] "RemoveContainer" containerID="9932d905fe41ce1a88718d80d4f2801a6bad16b507106d0a6a6ac13adab12e1e"
Jul 27 21:10:43.827100 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:10:43.827065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:10:58.232869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:10:58.232791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:11:06.799512 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:06.796041    2581 scope.go:117] "RemoveContainer" containerID="92ac42c28b1d9f20872944e3702a0e7e5375c1ca960e1c3219ae090dcb728b19"
Jul 27 21:11:09.634532 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"]
Jul 27 21:11:09.634893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634841    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88cf1bbb-7228-43ae-87fe-5446b27a7b06" containerName="main"
Jul 27 21:11:09.634893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634851    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88cf1bbb-7228-43ae-87fe-5446b27a7b06" containerName="main"
Jul 27 21:11:09.634893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634858    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfde08-685b-48b1-9035-2ab3e20039e3" containerName="revoke-keys"
Jul 27 21:11:09.634893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634864    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfde08-685b-48b1-9035-2ab3e20039e3" containerName="revoke-keys"
Jul 27 21:11:09.635027 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634909    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88cf1bbb-7228-43ae-87fe-5446b27a7b06" containerName="main"
Jul 27 21:11:09.635027 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.634921    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccfde08-685b-48b1-9035-2ab3e20039e3" containerName="revoke-keys"
Jul 27 21:11:09.639040 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.639022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.641271 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.641249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00b3a2\"/\"openshift-service-ca.crt\""
Jul 27 21:11:09.642069 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.642041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00b3a2\"/\"test-model-00b3a2-kserve-self-signed-certs\""
Jul 27 21:11:09.642171 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.642068    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00b3a2\"/\"kube-root-ca.crt\""
Jul 27 21:11:09.642171 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.642081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00b3a2\"/\"default-dockercfg-vwg46\""
Jul 27 21:11:09.647662 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.647637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"]
Jul 27 21:11:09.705315 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.705287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-home\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.705484 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.705328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kmp\" (UniqueName: \"kubernetes.io/projected/2e126d07-fe2d-4586-83e6-57231284983d-kube-api-access-n8kmp\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.705484 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.705444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-dshm\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.705603 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.705482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-model-cache\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.705603 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.705531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e126d07-fe2d-4586-83e6-57231284983d-tls-certs\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.705603 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.705582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-tmp-dir\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807072 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e126d07-fe2d-4586-83e6-57231284983d-tls-certs\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807072 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-tmp-dir\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807337 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-home\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807337 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kmp\" (UniqueName: \"kubernetes.io/projected/2e126d07-fe2d-4586-83e6-57231284983d-kube-api-access-n8kmp\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807473 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-dshm\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807523 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-model-cache\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807523 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-tmp-dir\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807645 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-home\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.807821 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.807802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-model-cache\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.809421 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.809405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-dshm\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.809614 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.809594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e126d07-fe2d-4586-83e6-57231284983d-tls-certs\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.818014 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.817992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kmp\" (UniqueName: \"kubernetes.io/projected/2e126d07-fe2d-4586-83e6-57231284983d-kube-api-access-n8kmp\") pod \"test-model-00b3a2-kserve-d67d6b797-dc74p\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") " pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:09.948375 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:09.948282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:10.081766 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:10.081730    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"]
Jul 27 21:11:10.084781 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:11:10.084752    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e126d07_fe2d_4586_83e6_57231284983d.slice/crio-7061149d721002cf510507a459bc2467a535dfbd2259694c4b51cef1c1802242 WatchSource:0}: Error finding container 7061149d721002cf510507a459bc2467a535dfbd2259694c4b51cef1c1802242: Status 404 returned error can't find the container with id 7061149d721002cf510507a459bc2467a535dfbd2259694c4b51cef1c1802242
Jul 27 21:11:10.235675 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:10.235593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p" event={"ID":"2e126d07-fe2d-4586-83e6-57231284983d","Type":"ContainerStarted","Data":"6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f"}
Jul 27 21:11:10.235675 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:10.235634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p" event={"ID":"2e126d07-fe2d-4586-83e6-57231284983d","Type":"ContainerStarted","Data":"7061149d721002cf510507a459bc2467a535dfbd2259694c4b51cef1c1802242"}
Jul 27 21:11:10.252882 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:10.252835    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p" podStartSLOduration=1.2528189250000001 podStartE2EDuration="1.252818925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:10.25186688 +0000 UTC m=+2164.132015585" watchObservedRunningTime="2026-07-27 21:11:10.252818925 +0000 UTC m=+2164.132967629"
Jul 27 21:11:14.131213 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:14.131177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:11:19.948878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:19.948839    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:19.949328 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:19.948932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:19.961202 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:19.961172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:20.291654 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:20.291578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:11:27.932344 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.932314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"]
Jul 27 21:11:27.935756 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.935738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:27.937986 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.937952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c7d2ad\"/\"kube-root-ca.crt\""
Jul 27 21:11:27.937986 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.937968    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c7d2ad\"/\"openshift-service-ca.crt\""
Jul 27 21:11:27.938714 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.938696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c7d2ad\"/\"default-dockercfg-bxz9q\""
Jul 27 21:11:27.938714 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.938709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c7d2ad\"/\"test-model-c7d2ad-kserve-self-signed-certs\""
Jul 27 21:11:27.943578 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.943537    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"]
Jul 27 21:11:27.967801 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.967770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-home\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:27.967953 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.967835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjfq5\" (UniqueName: \"kubernetes.io/projected/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-kube-api-access-fjfq5\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:27.967953 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.967871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-dshm\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:27.967953 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.967891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tls-certs\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:27.968108 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.967958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tmp-dir\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:27.968108 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:27.968012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-model-cache\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069014 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.068975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tmp-dir\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069195 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-model-cache\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069195 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-home\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069195 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjfq5\" (UniqueName: \"kubernetes.io/projected/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-kube-api-access-fjfq5\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069195 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-dshm\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069195 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tls-certs\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069445 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tmp-dir\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069503 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-home\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.069503 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.069482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-model-cache\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.071350 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.071321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-dshm\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.071608 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.071592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tls-certs\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.077265 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.077246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjfq5\" (UniqueName: \"kubernetes.io/projected/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-kube-api-access-fjfq5\") pod \"test-model-c7d2ad-kserve-5f4d4954c5-z86pk\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") " pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.247904 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.247822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:28.378606 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:28.378585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"]
Jul 27 21:11:28.381165 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:11:28.381140    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9105ff91_f7eb_4d96_8d5a_6cdf7791e4e9.slice/crio-47a0257aac127d012f8c44ffe31d3120f0d3eb18a76f20521e16eaf514307f4f WatchSource:0}: Error finding container 47a0257aac127d012f8c44ffe31d3120f0d3eb18a76f20521e16eaf514307f4f: Status 404 returned error can't find the container with id 47a0257aac127d012f8c44ffe31d3120f0d3eb18a76f20521e16eaf514307f4f
Jul 27 21:11:29.323326 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:29.323283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk" event={"ID":"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9","Type":"ContainerStarted","Data":"c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5"}
Jul 27 21:11:29.323741 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:29.323335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk" event={"ID":"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9","Type":"ContainerStarted","Data":"47a0257aac127d012f8c44ffe31d3120f0d3eb18a76f20521e16eaf514307f4f"}
Jul 27 21:11:29.357666 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:29.357618    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk" podStartSLOduration=2.357604729 podStartE2EDuration="2.357604729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:29.355968303 +0000 UTC m=+2183.236117018" watchObservedRunningTime="2026-07-27 21:11:29.357604729 +0000 UTC m=+2183.237753433"
Jul 27 21:11:34.931364 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:34.931329    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:11:38.248009 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:38.247967    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:38.248582 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:38.248029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:38.260232 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:38.260210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:38.376136 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:38.376112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:11:48.447567 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:11:48.447519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:12:10.283291 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:10.283251    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"]
Jul 27 21:12:11.229603 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.229572    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"]
Jul 27 21:12:11.347246 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.347214    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"]
Jul 27 21:12:11.350426 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.350403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:11.352489 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.352465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:12:11.352633 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.352488    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:12:11.357482 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.357447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"]
Jul 27 21:12:11.436134 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.436098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6153336-72af-4cfa-b290-fceb98888f6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9\" (UID: \"a6153336-72af-4cfa-b290-fceb98888f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:11.536892 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.536814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6153336-72af-4cfa-b290-fceb98888f6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9\" (UID: \"a6153336-72af-4cfa-b290-fceb98888f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:11.537406 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.537386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6153336-72af-4cfa-b290-fceb98888f6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9\" (UID: \"a6153336-72af-4cfa-b290-fceb98888f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:11.663409 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.663377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:11.800887 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:11.800811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"]
Jul 27 21:12:11.805535 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:12:11.805500    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6153336_72af_4cfa_b290_fceb98888f6a.slice/crio-d2148a5f7dbd1e288b614d39555ebfe5851dad42a40f46558797a242d592c37f WatchSource:0}: Error finding container d2148a5f7dbd1e288b614d39555ebfe5851dad42a40f46558797a242d592c37f: Status 404 returned error can't find the container with id d2148a5f7dbd1e288b614d39555ebfe5851dad42a40f46558797a242d592c37f
Jul 27 21:12:12.516113 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:12.516076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9" event={"ID":"a6153336-72af-4cfa-b290-fceb98888f6a","Type":"ContainerStarted","Data":"b1c1acd44b9661530aa21c1687b06adfcda9099ab50ba573f67ccc9ac9ff4e64"}
Jul 27 21:12:12.516511 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:12.516137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9" event={"ID":"a6153336-72af-4cfa-b290-fceb98888f6a","Type":"ContainerStarted","Data":"d2148a5f7dbd1e288b614d39555ebfe5851dad42a40f46558797a242d592c37f"}
Jul 27 21:12:12.531867 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:12.531819    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9" podStartSLOduration=1.531803976 podStartE2EDuration="1.531803976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:12.529956466 +0000 UTC m=+2226.410105169" watchObservedRunningTime="2026-07-27 21:12:12.531803976 +0000 UTC m=+2226.411952680"
Jul 27 21:12:13.521699 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:13.521667    2581 generic.go:356] "Generic (PLEG): container finished" podID="a6153336-72af-4cfa-b290-fceb98888f6a" containerID="b1c1acd44b9661530aa21c1687b06adfcda9099ab50ba573f67ccc9ac9ff4e64" exitCode=0
Jul 27 21:12:13.522117 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:13.521764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9" event={"ID":"a6153336-72af-4cfa-b290-fceb98888f6a","Type":"ContainerDied","Data":"b1c1acd44b9661530aa21c1687b06adfcda9099ab50ba573f67ccc9ac9ff4e64"}
Jul 27 21:12:14.645805 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:14.645780    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:14.762063 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:14.762035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6153336-72af-4cfa-b290-fceb98888f6a-maas-api-service-ca\") pod \"a6153336-72af-4cfa-b290-fceb98888f6a\" (UID: \"a6153336-72af-4cfa-b290-fceb98888f6a\") "
Jul 27 21:12:14.762398 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:14.762380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6153336-72af-4cfa-b290-fceb98888f6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6153336-72af-4cfa-b290-fceb98888f6a" (UID: "a6153336-72af-4cfa-b290-fceb98888f6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:12:14.833942 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:14.833870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:12:14.863635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:14.863612    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6153336-72af-4cfa-b290-fceb98888f6a-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:15.531667 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:15.531630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9" event={"ID":"a6153336-72af-4cfa-b290-fceb98888f6a","Type":"ContainerDied","Data":"d2148a5f7dbd1e288b614d39555ebfe5851dad42a40f46558797a242d592c37f"}
Jul 27 21:12:15.531843 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:15.531680    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2148a5f7dbd1e288b614d39555ebfe5851dad42a40f46558797a242d592c37f"
Jul 27 21:12:15.531843 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:15.531641    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"
Jul 27 21:12:25.302233 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.302189    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p" podUID="2e126d07-fe2d-4586-83e6-57231284983d" containerName="main" containerID="cri-o://6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f" gracePeriod=45
Jul 27 21:12:25.549070 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.549046    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:12:25.578423 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.578345    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e126d07-fe2d-4586-83e6-57231284983d" containerID="6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f" exitCode=0
Jul 27 21:12:25.578423 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.578389    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"
Jul 27 21:12:25.578423 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.578412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p" event={"ID":"2e126d07-fe2d-4586-83e6-57231284983d","Type":"ContainerDied","Data":"6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f"}
Jul 27 21:12:25.578729 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.578449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p" event={"ID":"2e126d07-fe2d-4586-83e6-57231284983d","Type":"ContainerDied","Data":"7061149d721002cf510507a459bc2467a535dfbd2259694c4b51cef1c1802242"}
Jul 27 21:12:25.578729 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.578471    2581 scope.go:117] "RemoveContainer" containerID="6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f"
Jul 27 21:12:25.588782 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.588759    2581 scope.go:117] "RemoveContainer" containerID="6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f"
Jul 27 21:12:25.589287 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:12:25.589260    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f\": container with ID starting with 6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f not found: ID does not exist" containerID="6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f"
Jul 27 21:12:25.589372 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.589300    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f"} err="failed to get container status \"6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f\": rpc error: code = NotFound desc = could not find container \"6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f\": container with ID starting with 6846fb85bc53004dcbd27681d29601cfdc7949fcbf7fe743232f9d41bd11bb0f not found: ID does not exist"
Jul 27 21:12:25.661795 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.661765    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e126d07-fe2d-4586-83e6-57231284983d-tls-certs\") pod \"2e126d07-fe2d-4586-83e6-57231284983d\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") "
Jul 27 21:12:25.661984 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.661818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-home\") pod \"2e126d07-fe2d-4586-83e6-57231284983d\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") "
Jul 27 21:12:25.661984 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.661870    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8kmp\" (UniqueName: \"kubernetes.io/projected/2e126d07-fe2d-4586-83e6-57231284983d-kube-api-access-n8kmp\") pod \"2e126d07-fe2d-4586-83e6-57231284983d\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") "
Jul 27 21:12:25.661984 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.661886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-model-cache\") pod \"2e126d07-fe2d-4586-83e6-57231284983d\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") "
Jul 27 21:12:25.661984 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.661918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-tmp-dir\") pod \"2e126d07-fe2d-4586-83e6-57231284983d\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") "
Jul 27 21:12:25.662186 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.662005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-dshm\") pod \"2e126d07-fe2d-4586-83e6-57231284983d\" (UID: \"2e126d07-fe2d-4586-83e6-57231284983d\") "
Jul 27 21:12:25.662186 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.662157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-home" (OuterVolumeSpecName: "home") pod "2e126d07-fe2d-4586-83e6-57231284983d" (UID: "2e126d07-fe2d-4586-83e6-57231284983d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:25.662273 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.662206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e126d07-fe2d-4586-83e6-57231284983d" (UID: "2e126d07-fe2d-4586-83e6-57231284983d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:25.662403 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.662379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e126d07-fe2d-4586-83e6-57231284983d" (UID: "2e126d07-fe2d-4586-83e6-57231284983d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:25.662532 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.662411    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:25.662532 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.662492    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:25.664316 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.664295    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-dshm" (OuterVolumeSpecName: "dshm") pod "2e126d07-fe2d-4586-83e6-57231284983d" (UID: "2e126d07-fe2d-4586-83e6-57231284983d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:25.664448 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.664432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e126d07-fe2d-4586-83e6-57231284983d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e126d07-fe2d-4586-83e6-57231284983d" (UID: "2e126d07-fe2d-4586-83e6-57231284983d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:12:25.664490 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.664467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e126d07-fe2d-4586-83e6-57231284983d-kube-api-access-n8kmp" (OuterVolumeSpecName: "kube-api-access-n8kmp") pod "2e126d07-fe2d-4586-83e6-57231284983d" (UID: "2e126d07-fe2d-4586-83e6-57231284983d"). InnerVolumeSpecName "kube-api-access-n8kmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:12:25.763590 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.763527    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8kmp\" (UniqueName: \"kubernetes.io/projected/2e126d07-fe2d-4586-83e6-57231284983d-kube-api-access-n8kmp\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:25.763590 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.763584    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:25.763590 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.763597    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e126d07-fe2d-4586-83e6-57231284983d-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:25.763814 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.763609    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e126d07-fe2d-4586-83e6-57231284983d-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:25.899827 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.899800    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"]
Jul 27 21:12:25.903931 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:25.903907    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-00b3a2/test-model-00b3a2-kserve-d67d6b797-dc74p"]
Jul 27 21:12:26.248299 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.248218    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk" podUID="9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" containerName="main" containerID="cri-o://c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5" gracePeriod=45
Jul 27 21:12:26.464709 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.464678    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"]
Jul 27 21:12:26.468222 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.468195    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-00b3a2-xvrl9"]
Jul 27 21:12:26.494457 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.494428    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:12:26.569636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569537    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjfq5\" (UniqueName: \"kubernetes.io/projected/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-kube-api-access-fjfq5\") pod \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") "
Jul 27 21:12:26.569636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-model-cache\") pod \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") "
Jul 27 21:12:26.569636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-dshm\") pod \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") "
Jul 27 21:12:26.569912 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569665    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tmp-dir\") pod \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") "
Jul 27 21:12:26.569912 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tls-certs\") pod \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") "
Jul 27 21:12:26.569912 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569747    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-home\") pod \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\" (UID: \"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9\") "
Jul 27 21:12:26.570054 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569922    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" (UID: "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:26.570054 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.569962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" (UID: "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:26.570144 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.570062    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:26.570144 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.570066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-home" (OuterVolumeSpecName: "home") pod "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" (UID: "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:26.571786 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.571764    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-dshm" (OuterVolumeSpecName: "dshm") pod "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" (UID: "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:12:26.571903 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.571857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" (UID: "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:12:26.571903 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.571894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-kube-api-access-fjfq5" (OuterVolumeSpecName: "kube-api-access-fjfq5") pod "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" (UID: "9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9"). InnerVolumeSpecName "kube-api-access-fjfq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:12:26.585729 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.585704    2581 generic.go:356] "Generic (PLEG): container finished" podID="9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" containerID="c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5" exitCode=0
Jul 27 21:12:26.585832 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.585746    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"
Jul 27 21:12:26.585832 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.585753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk" event={"ID":"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9","Type":"ContainerDied","Data":"c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5"}
Jul 27 21:12:26.585832 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.585787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk" event={"ID":"9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9","Type":"ContainerDied","Data":"47a0257aac127d012f8c44ffe31d3120f0d3eb18a76f20521e16eaf514307f4f"}
Jul 27 21:12:26.585832 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.585804    2581 scope.go:117] "RemoveContainer" containerID="c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5"
Jul 27 21:12:26.594228 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.594213    2581 scope.go:117] "RemoveContainer" containerID="c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5"
Jul 27 21:12:26.594475 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:12:26.594454    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5\": container with ID starting with c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5 not found: ID does not exist" containerID="c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5"
Jul 27 21:12:26.594539 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.594483    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5"} err="failed to get container status \"c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5\": rpc error: code = NotFound desc = could not find container \"c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5\": container with ID starting with c925ad44563e84b8869df8f19baa20564c0896f2ff6013fbcf16ab4db78100c5 not found: ID does not exist"
Jul 27 21:12:26.609475 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.609450    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"]
Jul 27 21:12:26.611412 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.611393    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c7d2ad/test-model-c7d2ad-kserve-5f4d4954c5-z86pk"]
Jul 27 21:12:26.670788 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.670766    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:26.670872 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.670806    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:26.670872 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.670815    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:26.670872 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.670825    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:26.670872 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.670834    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjfq5\" (UniqueName: \"kubernetes.io/projected/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9-kube-api-access-fjfq5\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:26.733516 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.733493    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e126d07-fe2d-4586-83e6-57231284983d" path="/var/lib/kubelet/pods/2e126d07-fe2d-4586-83e6-57231284983d/volumes"
Jul 27 21:12:26.733918 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.733905    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" path="/var/lib/kubelet/pods/9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9/volumes"
Jul 27 21:12:26.734181 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:26.734170    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6153336-72af-4cfa-b290-fceb98888f6a" path="/var/lib/kubelet/pods/a6153336-72af-4cfa-b290-fceb98888f6a/volumes"
Jul 27 21:12:27.440187 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:27.440149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:12:34.037264 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"]
Jul 27 21:12:34.037644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037568    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" containerName="main"
Jul 27 21:12:34.037644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037579    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" containerName="main"
Jul 27 21:12:34.037644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037591    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6153336-72af-4cfa-b290-fceb98888f6a" containerName="revoke-keys"
Jul 27 21:12:34.037644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037596    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6153336-72af-4cfa-b290-fceb98888f6a" containerName="revoke-keys"
Jul 27 21:12:34.037644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037611    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e126d07-fe2d-4586-83e6-57231284983d" containerName="main"
Jul 27 21:12:34.037644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037617    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e126d07-fe2d-4586-83e6-57231284983d" containerName="main"
Jul 27 21:12:34.037824 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037663    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9105ff91-f7eb-4d96-8d5a-6cdf7791e4e9" containerName="main"
Jul 27 21:12:34.037824 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037671    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6153336-72af-4cfa-b290-fceb98888f6a" containerName="revoke-keys"
Jul 27 21:12:34.037824 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.037677    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e126d07-fe2d-4586-83e6-57231284983d" containerName="main"
Jul 27 21:12:34.042149 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.042133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:34.045768 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.045748    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:12:34.045859 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.045800    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:12:34.062527 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.062502    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"]
Jul 27 21:12:34.130780 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.130741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s\" (UID: \"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:34.231400 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.231370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s\" (UID: \"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:34.231975 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.231956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s\" (UID: \"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:34.351792 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.351766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:34.477454 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.477430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"]
Jul 27 21:12:34.479542 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:12:34.479515    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffda9d5_1c00_4c0f_8d0e_c28db7e8e11a.slice/crio-2c5ea7fdfa18586f4f4bcfc8cb8306a0816c81294b30103682a0e6d5cd8d9166 WatchSource:0}: Error finding container 2c5ea7fdfa18586f4f4bcfc8cb8306a0816c81294b30103682a0e6d5cd8d9166: Status 404 returned error can't find the container with id 2c5ea7fdfa18586f4f4bcfc8cb8306a0816c81294b30103682a0e6d5cd8d9166
Jul 27 21:12:34.625031 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:34.624940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s" event={"ID":"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a","Type":"ContainerStarted","Data":"2c5ea7fdfa18586f4f4bcfc8cb8306a0816c81294b30103682a0e6d5cd8d9166"}
Jul 27 21:12:35.631062 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:35.631033    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a" containerID="f564c5ef0955c9a48667cb0fd59c2eba77d8f7c6074250c61cc1626d177ff72d" exitCode=0
Jul 27 21:12:35.631440 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:35.631086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s" event={"ID":"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a","Type":"ContainerDied","Data":"f564c5ef0955c9a48667cb0fd59c2eba77d8f7c6074250c61cc1626d177ff72d"}
Jul 27 21:12:36.751585 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:36.751563    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:36.853024 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:36.852995    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a-maas-api-service-ca\") pod \"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a\" (UID: \"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a\") "
Jul 27 21:12:36.853327 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:36.853308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a" (UID: "0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:12:36.954000 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:36.953931    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:12:37.641095 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:37.641065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"
Jul 27 21:12:37.641272 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:37.641097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s" event={"ID":"0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a","Type":"ContainerDied","Data":"2c5ea7fdfa18586f4f4bcfc8cb8306a0816c81294b30103682a0e6d5cd8d9166"}
Jul 27 21:12:37.641272 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:37.641139    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5ea7fdfa18586f4f4bcfc8cb8306a0816c81294b30103682a0e6d5cd8d9166"
Jul 27 21:12:47.337162 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:47.337124    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:12:49.101478 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:49.101445    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"]
Jul 27 21:12:49.104438 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:49.104416    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7d2ad-97g8s"]
Jul 27 21:12:50.733404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:12:50.733372    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a" path="/var/lib/kubelet/pods/0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a/volumes"
Jul 27 21:13:06.037820 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:06.037787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:13:06.823822 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:06.823795    2581 scope.go:117] "RemoveContainer" containerID="6359eebfb2d2c0dd7a01bd0f1bf1686057562836c7e1b972c8a0717ea27ed8c4"
Jul 27 21:13:08.206525 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.206488    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"]
Jul 27 21:13:08.206882 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.206829    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a" containerName="revoke-keys"
Jul 27 21:13:08.206882 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.206841    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a" containerName="revoke-keys"
Jul 27 21:13:08.206960 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.206903    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ffda9d5-1c00-4c0f-8d0e-c28db7e8e11a" containerName="revoke-keys"
Jul 27 21:13:08.211792 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.211776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.214121 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.214101    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:13:08.214232 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.214100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2mj6d\""
Jul 27 21:13:08.214843 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.214824    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:13:08.214952 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.214827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-965e57-serving-cert\""
Jul 27 21:13:08.219597 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.219577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"]
Jul 27 21:13:08.329841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.329811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18ca78-f720-446e-87f7-e64843416fba-openshift-service-ca\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.330005 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.329879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.330005 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.329915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczc8\" (UniqueName: \"kubernetes.io/projected/ad18ca78-f720-446e-87f7-e64843416fba-kube-api-access-rczc8\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.430957 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.430915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18ca78-f720-446e-87f7-e64843416fba-openshift-service-ca\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.431129 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.431001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.431129 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.431043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczc8\" (UniqueName: \"kubernetes.io/projected/ad18ca78-f720-446e-87f7-e64843416fba-kube-api-access-rczc8\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.431244 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:13:08.431148    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-serving-cert: secret "maas-api-e2e-api-b-965e57-serving-cert" not found
Jul 27 21:13:08.431244 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:13:08.431239    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls podName:ad18ca78-f720-446e-87f7-e64843416fba nodeName:}" failed. No retries permitted until 2026-07-27 21:13:08.931214679 +0000 UTC m=+2282.811363378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls") pod "maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" (UID: "ad18ca78-f720-446e-87f7-e64843416fba") : secret "maas-api-e2e-api-b-965e57-serving-cert" not found
Jul 27 21:13:08.431701 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.431684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18ca78-f720-446e-87f7-e64843416fba-openshift-service-ca\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.439704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.439684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczc8\" (UniqueName: \"kubernetes.io/projected/ad18ca78-f720-446e-87f7-e64843416fba-kube-api-access-rczc8\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.934944 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.934904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:08.937078 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:08.937054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls\") pod \"maas-api-e2e-api-b-965e57-89b4f645c-qgz7q\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:09.123307 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:09.123258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:09.250919 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:09.250887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"]
Jul 27 21:13:09.254259 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:13:09.254230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad18ca78_f720_446e_87f7_e64843416fba.slice/crio-108779659ab895ffb575bd35dc8b7a1daf73d6b1b41eed2ff8656bb334abf219 WatchSource:0}: Error finding container 108779659ab895ffb575bd35dc8b7a1daf73d6b1b41eed2ff8656bb334abf219: Status 404 returned error can't find the container with id 108779659ab895ffb575bd35dc8b7a1daf73d6b1b41eed2ff8656bb334abf219
Jul 27 21:13:09.255794 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:09.255774    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:13:09.792311 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:09.792266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" event={"ID":"ad18ca78-f720-446e-87f7-e64843416fba","Type":"ContainerStarted","Data":"108779659ab895ffb575bd35dc8b7a1daf73d6b1b41eed2ff8656bb334abf219"}
Jul 27 21:13:11.804763 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:11.804677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" event={"ID":"ad18ca78-f720-446e-87f7-e64843416fba","Type":"ContainerStarted","Data":"932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4"}
Jul 27 21:13:11.805117 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:11.804798    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:11.822985 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:11.822945    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" podStartSLOduration=1.6950663339999998 podStartE2EDuration="3.822925317s" totalImagesPullingTime="2.127858983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:13:11.821817673 +0000 UTC m=+2285.701966398" watchObservedRunningTime="2026-07-27 21:13:11.822925317 +0000 UTC m=+2285.703074020"
Jul 27 21:13:17.814247 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:17.814216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:13:20.589461 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.589426    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"]
Jul 27 21:13:20.592688 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.592665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:20.600857 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.600833    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"]
Jul 27 21:13:20.601912 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.601890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:13:20.638472 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.638435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304afb6f-baf7-4216-8619-ae9902d0e38c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf\" (UID: \"304afb6f-baf7-4216-8619-ae9902d0e38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:20.739763 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.739732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304afb6f-baf7-4216-8619-ae9902d0e38c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf\" (UID: \"304afb6f-baf7-4216-8619-ae9902d0e38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:20.740301 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.740281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304afb6f-baf7-4216-8619-ae9902d0e38c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf\" (UID: \"304afb6f-baf7-4216-8619-ae9902d0e38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:20.904274 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:20.904193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:21.029679 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:21.029599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"]
Jul 27 21:13:21.032004 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:13:21.031972    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304afb6f_baf7_4216_8619_ae9902d0e38c.slice/crio-e641ed88b59f9156125115131473000ddbf635ab45b4a303bc4daa7ffdd3cc52 WatchSource:0}: Error finding container e641ed88b59f9156125115131473000ddbf635ab45b4a303bc4daa7ffdd3cc52: Status 404 returned error can't find the container with id e641ed88b59f9156125115131473000ddbf635ab45b4a303bc4daa7ffdd3cc52
Jul 27 21:13:21.848841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:21.848807    2581 generic.go:356] "Generic (PLEG): container finished" podID="304afb6f-baf7-4216-8619-ae9902d0e38c" containerID="5d022a83ceaf5ef1f64e0c6472986a323896fb39e71f79634f9f7c16c7be6b36" exitCode=0
Jul 27 21:13:21.849181 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:21.848877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf" event={"ID":"304afb6f-baf7-4216-8619-ae9902d0e38c","Type":"ContainerDied","Data":"5d022a83ceaf5ef1f64e0c6472986a323896fb39e71f79634f9f7c16c7be6b36"}
Jul 27 21:13:21.849181 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:21.848915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf" event={"ID":"304afb6f-baf7-4216-8619-ae9902d0e38c","Type":"ContainerStarted","Data":"e641ed88b59f9156125115131473000ddbf635ab45b4a303bc4daa7ffdd3cc52"}
Jul 27 21:13:22.738322 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:22.738291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:13:22.977261 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:22.977235    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:23.074890 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:23.074792    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304afb6f-baf7-4216-8619-ae9902d0e38c-maas-api-service-ca\") pod \"304afb6f-baf7-4216-8619-ae9902d0e38c\" (UID: \"304afb6f-baf7-4216-8619-ae9902d0e38c\") "
Jul 27 21:13:23.075159 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:23.075135    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304afb6f-baf7-4216-8619-ae9902d0e38c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "304afb6f-baf7-4216-8619-ae9902d0e38c" (UID: "304afb6f-baf7-4216-8619-ae9902d0e38c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:13:23.176122 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:23.176088    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304afb6f-baf7-4216-8619-ae9902d0e38c-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:13:23.859145 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:23.859115    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"
Jul 27 21:13:23.859294 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:23.859154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf" event={"ID":"304afb6f-baf7-4216-8619-ae9902d0e38c","Type":"ContainerDied","Data":"e641ed88b59f9156125115131473000ddbf635ab45b4a303bc4daa7ffdd3cc52"}
Jul 27 21:13:23.859294 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:23.859199    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e641ed88b59f9156125115131473000ddbf635ab45b4a303bc4daa7ffdd3cc52"
Jul 27 21:13:40.139340 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:40.139304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:13:40.695809 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:40.695771    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"]
Jul 27 21:13:40.697946 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:40.697918    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f47ead-rgbqf"]
Jul 27 21:13:40.733878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:40.733849    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="304afb6f-baf7-4216-8619-ae9902d0e38c" path="/var/lib/kubelet/pods/304afb6f-baf7-4216-8619-ae9902d0e38c/volumes"
Jul 27 21:13:47.822576 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.822531    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"]
Jul 27 21:13:47.823004 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.822889    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="304afb6f-baf7-4216-8619-ae9902d0e38c" containerName="revoke-keys"
Jul 27 21:13:47.823004 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.822899    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="304afb6f-baf7-4216-8619-ae9902d0e38c" containerName="revoke-keys"
Jul 27 21:13:47.823004 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.822988    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="304afb6f-baf7-4216-8619-ae9902d0e38c" containerName="revoke-keys"
Jul 27 21:13:47.827276 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.827259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:47.829592 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.829561    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:13:47.833644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.833621    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"]
Jul 27 21:13:47.994705 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:47.994673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2585a64b-722d-4b84-9c2c-c01e17192737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-965e57-t4d4g\" (UID: \"2585a64b-722d-4b84-9c2c-c01e17192737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:48.096036 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.096002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2585a64b-722d-4b84-9c2c-c01e17192737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-965e57-t4d4g\" (UID: \"2585a64b-722d-4b84-9c2c-c01e17192737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:48.096673 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.096652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2585a64b-722d-4b84-9c2c-c01e17192737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-965e57-t4d4g\" (UID: \"2585a64b-722d-4b84-9c2c-c01e17192737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:48.138057 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.138013    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:48.264828 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.264772    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"]
Jul 27 21:13:48.267231 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:13:48.267204    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2585a64b_722d_4b84_9c2c_c01e17192737.slice/crio-aebb94346b26a63b9ecd9f538061485a77e6eb73e2de5cc49b7acdf2fc5a2b02 WatchSource:0}: Error finding container aebb94346b26a63b9ecd9f538061485a77e6eb73e2de5cc49b7acdf2fc5a2b02: Status 404 returned error can't find the container with id aebb94346b26a63b9ecd9f538061485a77e6eb73e2de5cc49b7acdf2fc5a2b02
Jul 27 21:13:48.976191 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.976157    2581 generic.go:356] "Generic (PLEG): container finished" podID="2585a64b-722d-4b84-9c2c-c01e17192737" containerID="5d2be71a4cd02ff73dc734822deeb29427a766c04aba2a41271d1fa7095981ee" exitCode=0
Jul 27 21:13:48.976616 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.976237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g" event={"ID":"2585a64b-722d-4b84-9c2c-c01e17192737","Type":"ContainerDied","Data":"5d2be71a4cd02ff73dc734822deeb29427a766c04aba2a41271d1fa7095981ee"}
Jul 27 21:13:48.976616 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:48.976285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g" event={"ID":"2585a64b-722d-4b84-9c2c-c01e17192737","Type":"ContainerStarted","Data":"aebb94346b26a63b9ecd9f538061485a77e6eb73e2de5cc49b7acdf2fc5a2b02"}
Jul 27 21:13:50.099661 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.099638    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:50.213716 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.213678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2585a64b-722d-4b84-9c2c-c01e17192737-maas-api-service-ca\") pod \"2585a64b-722d-4b84-9c2c-c01e17192737\" (UID: \"2585a64b-722d-4b84-9c2c-c01e17192737\") "
Jul 27 21:13:50.214064 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.214044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2585a64b-722d-4b84-9c2c-c01e17192737-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2585a64b-722d-4b84-9c2c-c01e17192737" (UID: "2585a64b-722d-4b84-9c2c-c01e17192737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:13:50.314692 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.314612    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2585a64b-722d-4b84-9c2c-c01e17192737-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:13:50.985911 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.985872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g" event={"ID":"2585a64b-722d-4b84-9c2c-c01e17192737","Type":"ContainerDied","Data":"aebb94346b26a63b9ecd9f538061485a77e6eb73e2de5cc49b7acdf2fc5a2b02"}
Jul 27 21:13:50.985911 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.985907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"
Jul 27 21:13:50.986093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:13:50.985931    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aebb94346b26a63b9ecd9f538061485a77e6eb73e2de5cc49b7acdf2fc5a2b02"
Jul 27 21:14:03.302565 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.302463    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"]
Jul 27 21:14:03.303014 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.302720    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" podUID="ad18ca78-f720-446e-87f7-e64843416fba" containerName="maas-api" containerID="cri-o://932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4" gracePeriod=30
Jul 27 21:14:03.542392 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.542369    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:14:03.631164 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.631131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls\") pod \"ad18ca78-f720-446e-87f7-e64843416fba\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") "
Jul 27 21:14:03.631347 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.631205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rczc8\" (UniqueName: \"kubernetes.io/projected/ad18ca78-f720-446e-87f7-e64843416fba-kube-api-access-rczc8\") pod \"ad18ca78-f720-446e-87f7-e64843416fba\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") "
Jul 27 21:14:03.631391 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.631366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18ca78-f720-446e-87f7-e64843416fba-openshift-service-ca\") pod \"ad18ca78-f720-446e-87f7-e64843416fba\" (UID: \"ad18ca78-f720-446e-87f7-e64843416fba\") "
Jul 27 21:14:03.631754 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.631731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad18ca78-f720-446e-87f7-e64843416fba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad18ca78-f720-446e-87f7-e64843416fba" (UID: "ad18ca78-f720-446e-87f7-e64843416fba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:14:03.633374 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.633355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad18ca78-f720-446e-87f7-e64843416fba" (UID: "ad18ca78-f720-446e-87f7-e64843416fba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:14:03.633455 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.633356    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad18ca78-f720-446e-87f7-e64843416fba-kube-api-access-rczc8" (OuterVolumeSpecName: "kube-api-access-rczc8") pod "ad18ca78-f720-446e-87f7-e64843416fba" (UID: "ad18ca78-f720-446e-87f7-e64843416fba"). InnerVolumeSpecName "kube-api-access-rczc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:14:03.732634 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.732598    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad18ca78-f720-446e-87f7-e64843416fba-maas-api-tls\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:14:03.732634 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.732626    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczc8\" (UniqueName: \"kubernetes.io/projected/ad18ca78-f720-446e-87f7-e64843416fba-kube-api-access-rczc8\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:14:03.732634 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:03.732636    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad18ca78-f720-446e-87f7-e64843416fba-openshift-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:14:04.045077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.044983    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad18ca78-f720-446e-87f7-e64843416fba" containerID="932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4" exitCode=0
Jul 27 21:14:04.045077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.045028    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"
Jul 27 21:14:04.045077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.045028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" event={"ID":"ad18ca78-f720-446e-87f7-e64843416fba","Type":"ContainerDied","Data":"932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4"}
Jul 27 21:14:04.045325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.045082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q" event={"ID":"ad18ca78-f720-446e-87f7-e64843416fba","Type":"ContainerDied","Data":"108779659ab895ffb575bd35dc8b7a1daf73d6b1b41eed2ff8656bb334abf219"}
Jul 27 21:14:04.045325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.045111    2581 scope.go:117] "RemoveContainer" containerID="932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4"
Jul 27 21:14:04.054825 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.054808    2581 scope.go:117] "RemoveContainer" containerID="932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4"
Jul 27 21:14:04.055085 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:14:04.055066    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4\": container with ID starting with 932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4 not found: ID does not exist" containerID="932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4"
Jul 27 21:14:04.055134 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.055094    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4"} err="failed to get container status \"932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4\": rpc error: code = NotFound desc = could not find container \"932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4\": container with ID starting with 932688b7b0b9bed47cfcb518949fc4a77b3ff57bb2b7082b1b2182e983ed66c4 not found: ID does not exist"
Jul 27 21:14:04.068085 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.068059    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"]
Jul 27 21:14:04.070321 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.070297    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-965e57-89b4f645c-qgz7q"]
Jul 27 21:14:04.733855 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:04.733822    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad18ca78-f720-446e-87f7-e64843416fba" path="/var/lib/kubelet/pods/ad18ca78-f720-446e-87f7-e64843416fba/volumes"
Jul 27 21:14:07.907758 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:07.907724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"]
Jul 27 21:14:07.910197 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:07.910176    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-965e57-t4d4g"]
Jul 27 21:14:07.937806 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:07.937782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:14:08.733835 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:08.733802    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2585a64b-722d-4b84-9c2c-c01e17192737" path="/var/lib/kubelet/pods/2585a64b-722d-4b84-9c2c-c01e17192737/volumes"
Jul 27 21:14:18.810372 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810329    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"]
Jul 27 21:14:18.810869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810696    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2585a64b-722d-4b84-9c2c-c01e17192737" containerName="revoke-keys"
Jul 27 21:14:18.810869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810708    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2585a64b-722d-4b84-9c2c-c01e17192737" containerName="revoke-keys"
Jul 27 21:14:18.810869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad18ca78-f720-446e-87f7-e64843416fba" containerName="maas-api"
Jul 27 21:14:18.810869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810731    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad18ca78-f720-446e-87f7-e64843416fba" containerName="maas-api"
Jul 27 21:14:18.810869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810785    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2585a64b-722d-4b84-9c2c-c01e17192737" containerName="revoke-keys"
Jul 27 21:14:18.810869 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.810795    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad18ca78-f720-446e-87f7-e64843416fba" containerName="maas-api"
Jul 27 21:14:18.813536 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.813519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:18.815936 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.815910    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-69909d-serving-cert\""
Jul 27 21:14:18.816071 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.816053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:14:18.816211 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.816192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:14:18.816758 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.816743    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2mj6d\""
Jul 27 21:14:18.823906 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.823883    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"]
Jul 27 21:14:18.963053 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.963017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-maas-api-tls\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:18.963224 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.963062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6fq\" (UniqueName: \"kubernetes.io/projected/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-kube-api-access-kh6fq\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:18.963224 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:18.963084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.064350 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.064256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-maas-api-tls\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.064350 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.064297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6fq\" (UniqueName: \"kubernetes.io/projected/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-kube-api-access-kh6fq\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.064350 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.064317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.065054 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.065036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.066685 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.066666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-maas-api-tls\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.072384 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.072363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6fq\" (UniqueName: \"kubernetes.io/projected/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-kube-api-access-kh6fq\") pod \"maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.124627 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.124603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:19.254535 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:19.254380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"]
Jul 27 21:14:19.256806 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:14:19.256777    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c26ab1_0f9a_484d_aee6_be7cda989d3e.slice/crio-c4559ebc6d0510a959408bdac92bf99010dce01fdb14c36fea0abf2c4de2c63c WatchSource:0}: Error finding container c4559ebc6d0510a959408bdac92bf99010dce01fdb14c36fea0abf2c4de2c63c: Status 404 returned error can't find the container with id c4559ebc6d0510a959408bdac92bf99010dce01fdb14c36fea0abf2c4de2c63c
Jul 27 21:14:20.123936 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:20.123896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw" event={"ID":"b9c26ab1-0f9a-484d-aee6-be7cda989d3e","Type":"ContainerStarted","Data":"1c6c55e57b8dc362d8209079f01c313cbd5c7b9d354b5dfe537edf1125c12221"}
Jul 27 21:14:20.124329 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:20.123980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw" event={"ID":"b9c26ab1-0f9a-484d-aee6-be7cda989d3e","Type":"ContainerStarted","Data":"c4559ebc6d0510a959408bdac92bf99010dce01fdb14c36fea0abf2c4de2c63c"}
Jul 27 21:14:20.124329 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:20.124035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:20.147965 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:20.147910    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw" podStartSLOduration=1.691782436 podStartE2EDuration="2.147894475s" totalImagesPullingTime="456.112039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:20.146885177 +0000 UTC m=+2354.027033881" watchObservedRunningTime="2026-07-27 21:14:20.147894475 +0000 UTC m=+2354.028043170"
Jul 27 21:14:24.435788 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:24.435755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:14:26.133783 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.133756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:14:26.801294 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.801263    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"]
Jul 27 21:14:26.804740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.804718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:26.807879 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.807855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ba2511-serving-cert\""
Jul 27 21:14:26.819698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.819673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"]
Jul 27 21:14:26.940884 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.940854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd995\" (UniqueName: \"kubernetes.io/projected/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-kube-api-access-nd995\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:26.941034 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.940899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:26.941034 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:26.940982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.041968 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:27.041935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.042123 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:27.041981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd995\" (UniqueName: \"kubernetes.io/projected/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-kube-api-access-nd995\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.042123 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:27.042010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.042123 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:14:27.042100    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-serving-cert: secret "maas-api-e2e-shared-b-ba2511-serving-cert" not found
Jul 27 21:14:27.042227 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:14:27.042162    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls podName:7553fdcf-b578-4dfd-9404-d6ed3f4557b2 nodeName:}" failed. No retries permitted until 2026-07-27 21:14:27.542144963 +0000 UTC m=+2361.422293646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls") pod "maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" (UID: "7553fdcf-b578-4dfd-9404-d6ed3f4557b2") : secret "maas-api-e2e-shared-b-ba2511-serving-cert" not found
Jul 27 21:14:27.042598 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:27.042580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.052447 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:27.052393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd995\" (UniqueName: \"kubernetes.io/projected/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-kube-api-access-nd995\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.545700 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:27.545666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:27.546077 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:14:27.545818    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-serving-cert: secret "maas-api-e2e-shared-b-ba2511-serving-cert" not found
Jul 27 21:14:27.546077 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:14:27.545894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls podName:7553fdcf-b578-4dfd-9404-d6ed3f4557b2 nodeName:}" failed. No retries permitted until 2026-07-27 21:14:28.545877074 +0000 UTC m=+2362.426025756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls") pod "maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" (UID: "7553fdcf-b578-4dfd-9404-d6ed3f4557b2") : secret "maas-api-e2e-shared-b-ba2511-serving-cert" not found
Jul 27 21:14:28.554585 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:28.554535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:28.556953 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:28.556930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:28.615836 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:28.615809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:28.765413 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:28.765388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"]
Jul 27 21:14:28.768178 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:14:28.768150    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7553fdcf_b578_4dfd_9404_d6ed3f4557b2.slice/crio-f7c92e122a3fb85992eb73acaf13fac139b8315e0f4445a970e1154ff7fc40a5 WatchSource:0}: Error finding container f7c92e122a3fb85992eb73acaf13fac139b8315e0f4445a970e1154ff7fc40a5: Status 404 returned error can't find the container with id f7c92e122a3fb85992eb73acaf13fac139b8315e0f4445a970e1154ff7fc40a5
Jul 27 21:14:29.169100 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:29.169074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" event={"ID":"7553fdcf-b578-4dfd-9404-d6ed3f4557b2","Type":"ContainerStarted","Data":"cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc"}
Jul 27 21:14:29.169229 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:29.169129    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:29.169229 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:29.169151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" event={"ID":"7553fdcf-b578-4dfd-9404-d6ed3f4557b2","Type":"ContainerStarted","Data":"f7c92e122a3fb85992eb73acaf13fac139b8315e0f4445a970e1154ff7fc40a5"}
Jul 27 21:14:29.188050 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:29.188006    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" podStartSLOduration=2.863022129 podStartE2EDuration="3.187989132s" totalImagesPullingTime="324.967003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:29.187172605 +0000 UTC m=+2363.067321310" watchObservedRunningTime="2026-07-27 21:14:29.187989132 +0000 UTC m=+2363.068137836"
Jul 27 21:14:35.179310 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:35.179277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:14:40.741726 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:40.741693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:14:45.162928 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.162897    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"]
Jul 27 21:14:45.166531 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.166508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.171311 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.171242    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"kube-root-ca.crt\""
Jul 27 21:14:45.171311 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.171252    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"openshift-service-ca.crt\""
Jul 27 21:14:45.171311 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.171279    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"auth-test-model-69909d-kserve-self-signed-certs\""
Jul 27 21:14:45.171618 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.171283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"default-dockercfg-lwsxj\""
Jul 27 21:14:45.181826 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.181799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"]
Jul 27 21:14:45.191786 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.191761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rvf\" (UniqueName: \"kubernetes.io/projected/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-kube-api-access-n8rvf\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.191924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.191797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-home\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.191924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.191822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tls-certs\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.191924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.191897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-model-cache\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.192057 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.191945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-dshm\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.192057 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.192022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tmp-dir\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293359 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tmp-dir\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293359 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rvf\" (UniqueName: \"kubernetes.io/projected/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-kube-api-access-n8rvf\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293671 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-home\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293671 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tls-certs\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293671 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-model-cache\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293671 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-dshm\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293879 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tmp-dir\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293879 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-home\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.293961 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.293926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-model-cache\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.295702 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.295681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-dshm\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.296020 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.296005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tls-certs\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.301455 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.301433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rvf\" (UniqueName: \"kubernetes.io/projected/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-kube-api-access-n8rvf\") pod \"auth-test-model-69909d-kserve-687b54f8c-9cxx9\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") " pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.477381 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.477293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:45.608541 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:45.608516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"]
Jul 27 21:14:45.610811 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:14:45.610785    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95dc8b6_8c53_404b_ac0d_b64b7cb524fa.slice/crio-6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475 WatchSource:0}: Error finding container 6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475: Status 404 returned error can't find the container with id 6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475
Jul 27 21:14:46.247980 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:46.247946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9" event={"ID":"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa","Type":"ContainerStarted","Data":"c8836f75153c19b7522c4555755f8d974bcf3cee11a4d1c91cdcfdd082a2f02e"}
Jul 27 21:14:46.247980 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:46.247990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9" event={"ID":"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa","Type":"ContainerStarted","Data":"6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475"}
Jul 27 21:14:46.267735 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:46.267683    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9" podStartSLOduration=1.267666916 podStartE2EDuration="1.267666916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:46.266097804 +0000 UTC m=+2380.146246519" watchObservedRunningTime="2026-07-27 21:14:46.267666916 +0000 UTC m=+2380.147815621"
Jul 27 21:14:50.353087 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:50.353055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:14:55.477666 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:55.477631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:55.478135 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:55.477702    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:55.489698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:55.489677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:14:56.309332 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:14:56.309302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:15:00.208233 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.208198    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"]
Jul 27 21:15:00.211685 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.211668    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"]
Jul 27 21:15:00.211797 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.211779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"
Jul 27 21:15:00.214735 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.214715    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"]
Jul 27 21:15:00.214837 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.214758    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:15:00.214837 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.214814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"
Jul 27 21:15:00.217679 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.217662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"
Jul 27 21:15:00.222236 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.222221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"
Jul 27 21:15:00.229388 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.229371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"
Jul 27 21:15:00.229611 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.229439    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"]
Jul 27 21:15:00.239360 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.238742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"]
Jul 27 21:15:00.243751 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.243725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"]
Jul 27 21:15:00.269437 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.269414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"
Jul 27 21:15:00.402730 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.402701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"]
Jul 27 21:15:00.404979 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:15:00.404957    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d97c5a1_4c4c_4d92_9d0d_0c25d76cf508.slice/crio-f39c845758356059d94f4f88419a8359b64d4ae06de82aca23184f46ebdd16d3 WatchSource:0}: Error finding container f39c845758356059d94f4f88419a8359b64d4ae06de82aca23184f46ebdd16d3: Status 404 returned error can't find the container with id f39c845758356059d94f4f88419a8359b64d4ae06de82aca23184f46ebdd16d3
Jul 27 21:15:00.435398 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.435374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"]
Jul 27 21:15:00.440276 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:15:00.440251    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b946c3_cd56_484b_b7eb_7427d27aa13b.slice/crio-3ae88eb0da49022b91c7253f1d8bebc485229306476f15040df4b632b5e9497c WatchSource:0}: Error finding container 3ae88eb0da49022b91c7253f1d8bebc485229306476f15040df4b632b5e9497c: Status 404 returned error can't find the container with id 3ae88eb0da49022b91c7253f1d8bebc485229306476f15040df4b632b5e9497c
Jul 27 21:15:00.459601 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:00.459529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"]
Jul 27 21:15:00.461684 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:15:00.461660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d18851_5efb_4185_bc18_7ba36328e49a.slice/crio-a099cbd067b5adcd50a4a82fb1560d31ecacd9bdc4ef6cb91aff9f2d8ec27b46 WatchSource:0}: Error finding container a099cbd067b5adcd50a4a82fb1560d31ecacd9bdc4ef6cb91aff9f2d8ec27b46: Status 404 returned error can't find the container with id a099cbd067b5adcd50a4a82fb1560d31ecacd9bdc4ef6cb91aff9f2d8ec27b46
Jul 27 21:15:01.324698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.324661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7" event={"ID":"54d18851-5efb-4185-bc18-7ba36328e49a","Type":"ContainerStarted","Data":"1b729b4c636f4e33f37421fe3bb953c1447735bc6915f1a9b5a8692b45220146"}
Jul 27 21:15:01.325093 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.324722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7" event={"ID":"54d18851-5efb-4185-bc18-7ba36328e49a","Type":"ContainerStarted","Data":"a099cbd067b5adcd50a4a82fb1560d31ecacd9bdc4ef6cb91aff9f2d8ec27b46"}
Jul 27 21:15:01.326124 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.326103    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508" containerID="9c7a19be7d7141f0fbde979aee2f6003d7f4a51d44940dbffbcf0b34af3983cb" exitCode=0
Jul 27 21:15:01.326202 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.326186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc" event={"ID":"5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508","Type":"ContainerDied","Data":"9c7a19be7d7141f0fbde979aee2f6003d7f4a51d44940dbffbcf0b34af3983cb"}
Jul 27 21:15:01.326254 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.326231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc" event={"ID":"5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508","Type":"ContainerStarted","Data":"f39c845758356059d94f4f88419a8359b64d4ae06de82aca23184f46ebdd16d3"}
Jul 27 21:15:01.327559 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.327526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6" event={"ID":"44b946c3-cd56-484b-b7eb-7427d27aa13b","Type":"ContainerStarted","Data":"15d30c9a34876b9dac6477c901f11ee4846bfc11b3541bcf6f22ea39387600f5"}
Jul 27 21:15:01.327675 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.327577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6" event={"ID":"44b946c3-cd56-484b-b7eb-7427d27aa13b","Type":"ContainerStarted","Data":"3ae88eb0da49022b91c7253f1d8bebc485229306476f15040df4b632b5e9497c"}
Jul 27 21:15:01.342299 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.342241    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7" podStartSLOduration=1.34222131 podStartE2EDuration="1.34222131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:01.339926945 +0000 UTC m=+2395.220075649" watchObservedRunningTime="2026-07-27 21:15:01.34222131 +0000 UTC m=+2395.222370015"
Jul 27 21:15:01.373673 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:01.373619    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6" podStartSLOduration=1.373602163 podStartE2EDuration="1.373602163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:01.371993188 +0000 UTC m=+2395.252141928" watchObservedRunningTime="2026-07-27 21:15:01.373602163 +0000 UTC m=+2395.253750866"
Jul 27 21:15:02.334207 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:02.334177    2581 generic.go:356] "Generic (PLEG): container finished" podID="44b946c3-cd56-484b-b7eb-7427d27aa13b" containerID="15d30c9a34876b9dac6477c901f11ee4846bfc11b3541bcf6f22ea39387600f5" exitCode=0
Jul 27 21:15:02.334639 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:02.334265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6" event={"ID":"44b946c3-cd56-484b-b7eb-7427d27aa13b","Type":"ContainerDied","Data":"15d30c9a34876b9dac6477c901f11ee4846bfc11b3541bcf6f22ea39387600f5"}
Jul 27 21:15:02.335605 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:02.335586    2581 generic.go:356] "Generic (PLEG): container finished" podID="54d18851-5efb-4185-bc18-7ba36328e49a" containerID="1b729b4c636f4e33f37421fe3bb953c1447735bc6915f1a9b5a8692b45220146" exitCode=0
Jul 27 21:15:02.335683 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:02.335665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7" event={"ID":"54d18851-5efb-4185-bc18-7ba36328e49a","Type":"ContainerDied","Data":"1b729b4c636f4e33f37421fe3bb953c1447735bc6915f1a9b5a8692b45220146"}
Jul 27 21:15:02.458977 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:02.458953    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"
Jul 27 21:15:03.235142 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.235111    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"]
Jul 27 21:15:03.235693 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.235672    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508" containerName="cleanup"
Jul 27 21:15:03.235693 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.235692    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508" containerName="cleanup"
Jul 27 21:15:03.235852 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.235792    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508" containerName="cleanup"
Jul 27 21:15:03.239008 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.238990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.241804 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.241778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"kube-root-ca.crt\""
Jul 27 21:15:03.241804 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.241799    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"openshift-service-ca.crt\""
Jul 27 21:15:03.242680 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.242659    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"default-dockercfg-6qmtm\""
Jul 27 21:15:03.242787 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.242702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"auth-test-model-ba2511-kserve-self-signed-certs\""
Jul 27 21:15:03.250146 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.250122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"]
Jul 27 21:15:03.342247 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.342217    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"
Jul 27 21:15:03.342686 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.342274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc" event={"ID":"5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508","Type":"ContainerDied","Data":"f39c845758356059d94f4f88419a8359b64d4ae06de82aca23184f46ebdd16d3"}
Jul 27 21:15:03.342686 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.342338    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39c845758356059d94f4f88419a8359b64d4ae06de82aca23184f46ebdd16d3"
Jul 27 21:15:03.355113 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.355076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-home\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.355266 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.355115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af53d76-a48c-4319-9e29-d983fcdd829f-tls-certs\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.355266 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.355142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-dshm\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.355266 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.355249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-model-cache\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.355380 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.355324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lgq\" (UniqueName: \"kubernetes.io/projected/8af53d76-a48c-4319-9e29-d983fcdd829f-kube-api-access-p2lgq\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.355456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.355438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-tmp-dir\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456225 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-home\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456377 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af53d76-a48c-4319-9e29-d983fcdd829f-tls-certs\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456377 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-dshm\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456377 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-model-cache\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456377 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lgq\" (UniqueName: \"kubernetes.io/projected/8af53d76-a48c-4319-9e29-d983fcdd829f-kube-api-access-p2lgq\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456644 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-tmp-dir\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.456704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.456659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-home\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.457138 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.457006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-tmp-dir\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.457194 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.457140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-model-cache\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.459076 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.459044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af53d76-a48c-4319-9e29-d983fcdd829f-tls-certs\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.459217 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.459196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-dshm\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.465079 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.465058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lgq\" (UniqueName: \"kubernetes.io/projected/8af53d76-a48c-4319-9e29-d983fcdd829f-kube-api-access-p2lgq\") pod \"auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") " pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.490381 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.490327    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"
Jul 27 21:15:03.492305 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.492280    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"
Jul 27 21:15:03.549324 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.549287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:03.677704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:03.677678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"]
Jul 27 21:15:03.679348 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:15:03.679322    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8af53d76_a48c_4319_9e29_d983fcdd829f.slice/crio-8ee8a744900c69a2f06083afc6bf58ea8b0f60fdcac82e74224c088367a0b6b4 WatchSource:0}: Error finding container 8ee8a744900c69a2f06083afc6bf58ea8b0f60fdcac82e74224c088367a0b6b4: Status 404 returned error can't find the container with id 8ee8a744900c69a2f06083afc6bf58ea8b0f60fdcac82e74224c088367a0b6b4
Jul 27 21:15:04.349007 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.348971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" event={"ID":"8af53d76-a48c-4319-9e29-d983fcdd829f","Type":"ContainerStarted","Data":"42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9"}
Jul 27 21:15:04.349417 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.349021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" event={"ID":"8af53d76-a48c-4319-9e29-d983fcdd829f","Type":"ContainerStarted","Data":"8ee8a744900c69a2f06083afc6bf58ea8b0f60fdcac82e74224c088367a0b6b4"}
Jul 27 21:15:04.350215 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.350192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6"
Jul 27 21:15:04.350345 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.350223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-bh5v6" event={"ID":"44b946c3-cd56-484b-b7eb-7427d27aa13b","Type":"ContainerDied","Data":"3ae88eb0da49022b91c7253f1d8bebc485229306476f15040df4b632b5e9497c"}
Jul 27 21:15:04.350345 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.350257    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae88eb0da49022b91c7253f1d8bebc485229306476f15040df4b632b5e9497c"
Jul 27 21:15:04.351404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.351389    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"
Jul 27 21:15:04.351507 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.351385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7" event={"ID":"54d18851-5efb-4185-bc18-7ba36328e49a","Type":"ContainerDied","Data":"a099cbd067b5adcd50a4a82fb1560d31ecacd9bdc4ef6cb91aff9f2d8ec27b46"}
Jul 27 21:15:04.351568 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.351506    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a099cbd067b5adcd50a4a82fb1560d31ecacd9bdc4ef6cb91aff9f2d8ec27b46"
Jul 27 21:15:04.366699 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:04.366660    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" podStartSLOduration=1.366647003 podStartE2EDuration="1.366647003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:04.365356178 +0000 UTC m=+2398.245504882" watchObservedRunningTime="2026-07-27 21:15:04.366647003 +0000 UTC m=+2398.246795707"
Jul 27 21:15:06.733508 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:06.733478    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:15:06.737263 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:06.737234    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:15:06.846376 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:06.846334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:15:06.859796 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:06.859767    2581 scope.go:117] "RemoveContainer" containerID="eb1ece6cbfd1a521078b4d52c94c1d08bc95aca3092954af09adaef3f23ae7a6"
Jul 27 21:15:13.550404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:13.550374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:13.550783 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:13.550435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:13.562667 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:13.562642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:14.409965 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:14.409936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:15:32.842470 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:32.842434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:15:53.737267 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:15:53.737223    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:16:11.537059 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:16:11.537023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:16:35.041848 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:16:35.041813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:16:52.941981 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:16:52.941949    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:17:20.055925 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:17:20.055841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:17:32.952450 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:17:32.952414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:17:54.243647 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:17:54.243611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:18:12.748287 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:18:12.748251    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:18:39.343900 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:18:39.343811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:18:52.657120 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:18:52.657016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:19:06.907497 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:06.907463    2581 scope.go:117] "RemoveContainer" containerID="f564c5ef0955c9a48667cb0fd59c2eba77d8f7c6074250c61cc1626d177ff72d"
Jul 27 21:19:06.915384 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:06.915365    2581 scope.go:117] "RemoveContainer" containerID="b1c1acd44b9661530aa21c1687b06adfcda9099ab50ba573f67ccc9ac9ff4e64"
Jul 27 21:19:13.244669 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:13.244639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:19:18.465597 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:18.465531    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"]
Jul 27 21:19:19.311808 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:19.311778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"]
Jul 27 21:19:31.570975 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:31.570940    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:19:33.483877 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.483842    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9" podUID="d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" containerName="main" containerID="cri-o://c8836f75153c19b7522c4555755f8d974bcf3cee11a4d1c91cdcfdd082a2f02e" gracePeriod=45
Jul 27 21:19:33.554632 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:19:33.554601    2581 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95dc8b6_8c53_404b_ac0d_b64b7cb524fa.slice/crio-6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475/memory.min": read /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95dc8b6_8c53_404b_ac0d_b64b7cb524fa.slice/crio-6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475/memory.min: no such device
Jul 27 21:19:33.654019 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.653986    2581 generic.go:356] "Generic (PLEG): container finished" podID="d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" containerID="c8836f75153c19b7522c4555755f8d974bcf3cee11a4d1c91cdcfdd082a2f02e" exitCode=0
Jul 27 21:19:33.654138 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.654060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9" event={"ID":"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa","Type":"ContainerDied","Data":"c8836f75153c19b7522c4555755f8d974bcf3cee11a4d1c91cdcfdd082a2f02e"}
Jul 27 21:19:33.728484 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.728461    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:19:33.881376 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-dshm\") pod \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") "
Jul 27 21:19:33.881635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8rvf\" (UniqueName: \"kubernetes.io/projected/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-kube-api-access-n8rvf\") pod \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") "
Jul 27 21:19:33.881635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tmp-dir\") pod \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") "
Jul 27 21:19:33.881635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-model-cache\") pod \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") "
Jul 27 21:19:33.881635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-home\") pod \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") "
Jul 27 21:19:33.881635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tls-certs\") pod \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\" (UID: \"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa\") "
Jul 27 21:19:33.881905 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" (UID: "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:33.881905 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" (UID: "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:33.881905 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881885    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:33.881905 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-home" (OuterVolumeSpecName: "home") pod "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" (UID: "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:33.881905 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.881900    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:33.883634 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.883607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-dshm" (OuterVolumeSpecName: "dshm") pod "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" (UID: "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:33.883752 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.883666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" (UID: "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:19:33.883752 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.883689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-kube-api-access-n8rvf" (OuterVolumeSpecName: "kube-api-access-n8rvf") pod "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" (UID: "d95dc8b6-8c53-404b-ac0d-b64b7cb524fa"). InnerVolumeSpecName "kube-api-access-n8rvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:33.982436 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.982397    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rvf\" (UniqueName: \"kubernetes.io/projected/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-kube-api-access-n8rvf\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:33.982436 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.982430    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:33.982659 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.982445    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:33.982659 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:33.982457    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.330584 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.330463    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" podUID="8af53d76-a48c-4319-9e29-d983fcdd829f" containerName="main" containerID="cri-o://42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9" gracePeriod=45
Jul 27 21:19:34.399629 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.399582    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" podUID="8af53d76-a48c-4319-9e29-d983fcdd829f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 27 21:19:34.574993 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.574972    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:19:34.660166 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.660143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"
Jul 27 21:19:34.660333 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.660186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9" event={"ID":"d95dc8b6-8c53-404b-ac0d-b64b7cb524fa","Type":"ContainerDied","Data":"6c774d233514fbe81f8a4e5e341b7a79f76b892498da16cd42c5dac6ae810475"}
Jul 27 21:19:34.660333 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.660237    2581 scope.go:117] "RemoveContainer" containerID="c8836f75153c19b7522c4555755f8d974bcf3cee11a4d1c91cdcfdd082a2f02e"
Jul 27 21:19:34.661696 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.661673    2581 generic.go:356] "Generic (PLEG): container finished" podID="8af53d76-a48c-4319-9e29-d983fcdd829f" containerID="42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9" exitCode=0
Jul 27 21:19:34.661805 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.661709    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"
Jul 27 21:19:34.661805 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.661699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" event={"ID":"8af53d76-a48c-4319-9e29-d983fcdd829f","Type":"ContainerDied","Data":"42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9"}
Jul 27 21:19:34.661805 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.661779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk" event={"ID":"8af53d76-a48c-4319-9e29-d983fcdd829f","Type":"ContainerDied","Data":"8ee8a744900c69a2f06083afc6bf58ea8b0f60fdcac82e74224c088367a0b6b4"}
Jul 27 21:19:34.669367 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.669349    2581 scope.go:117] "RemoveContainer" containerID="42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9"
Jul 27 21:19:34.678475 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.678459    2581 scope.go:117] "RemoveContainer" containerID="42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9"
Jul 27 21:19:34.678847 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:19:34.678828    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9\": container with ID starting with 42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9 not found: ID does not exist" containerID="42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9"
Jul 27 21:19:34.678909 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.678854    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9"} err="failed to get container status \"42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9\": rpc error: code = NotFound desc = could not find container \"42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9\": container with ID starting with 42c7c8cd6d49a8af7c1429c7881505b874513474e10cc19187edd7d8982c0cd9 not found: ID does not exist"
Jul 27 21:19:34.688004 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.687983    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"]
Jul 27 21:19:34.689179 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-tmp-dir\") pod \"8af53d76-a48c-4319-9e29-d983fcdd829f\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") "
Jul 27 21:19:34.689297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689215    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-dshm\") pod \"8af53d76-a48c-4319-9e29-d983fcdd829f\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") "
Jul 27 21:19:34.689297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af53d76-a48c-4319-9e29-d983fcdd829f-tls-certs\") pod \"8af53d76-a48c-4319-9e29-d983fcdd829f\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") "
Jul 27 21:19:34.689297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-model-cache\") pod \"8af53d76-a48c-4319-9e29-d983fcdd829f\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") "
Jul 27 21:19:34.689461 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2lgq\" (UniqueName: \"kubernetes.io/projected/8af53d76-a48c-4319-9e29-d983fcdd829f-kube-api-access-p2lgq\") pod \"8af53d76-a48c-4319-9e29-d983fcdd829f\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") "
Jul 27 21:19:34.689461 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689406    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-home\") pod \"8af53d76-a48c-4319-9e29-d983fcdd829f\" (UID: \"8af53d76-a48c-4319-9e29-d983fcdd829f\") "
Jul 27 21:19:34.689461 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689440    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8af53d76-a48c-4319-9e29-d983fcdd829f" (UID: "8af53d76-a48c-4319-9e29-d983fcdd829f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:34.689630 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8af53d76-a48c-4319-9e29-d983fcdd829f" (UID: "8af53d76-a48c-4319-9e29-d983fcdd829f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:34.689726 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-home" (OuterVolumeSpecName: "home") pod "8af53d76-a48c-4319-9e29-d983fcdd829f" (UID: "8af53d76-a48c-4319-9e29-d983fcdd829f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:34.689774 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689727    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.689774 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.689747    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.690400 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.690381    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/auth-test-model-69909d-kserve-687b54f8c-9cxx9"]
Jul 27 21:19:34.691439 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.691414    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af53d76-a48c-4319-9e29-d983fcdd829f-kube-api-access-p2lgq" (OuterVolumeSpecName: "kube-api-access-p2lgq") pod "8af53d76-a48c-4319-9e29-d983fcdd829f" (UID: "8af53d76-a48c-4319-9e29-d983fcdd829f"). InnerVolumeSpecName "kube-api-access-p2lgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:34.691489 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.691470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af53d76-a48c-4319-9e29-d983fcdd829f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8af53d76-a48c-4319-9e29-d983fcdd829f" (UID: "8af53d76-a48c-4319-9e29-d983fcdd829f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:19:34.691489 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.691476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-dshm" (OuterVolumeSpecName: "dshm") pod "8af53d76-a48c-4319-9e29-d983fcdd829f" (UID: "8af53d76-a48c-4319-9e29-d983fcdd829f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:19:34.734518 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.734493    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" path="/var/lib/kubelet/pods/d95dc8b6-8c53-404b-ac0d-b64b7cb524fa/volumes"
Jul 27 21:19:34.790456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.790428    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.790456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.790454    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8af53d76-a48c-4319-9e29-d983fcdd829f-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.790456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.790462    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8af53d76-a48c-4319-9e29-d983fcdd829f-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.790715 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.790471    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lgq\" (UniqueName: \"kubernetes.io/projected/8af53d76-a48c-4319-9e29-d983fcdd829f-kube-api-access-p2lgq\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:19:34.980962 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.980885    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"]
Jul 27 21:19:34.997921 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:34.993987    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/auth-test-model-ba2511-kserve-6d4ccd6459-zq6fk"]
Jul 27 21:19:36.733654 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:36.733622    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8af53d76-a48c-4319-9e29-d983fcdd829f" path="/var/lib/kubelet/pods/8af53d76-a48c-4319-9e29-d983fcdd829f/volumes"
Jul 27 21:19:56.234763 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.234732    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"]
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b946c3-cd56-484b-b7eb-7427d27aa13b" containerName="cleanup"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b946c3-cd56-484b-b7eb-7427d27aa13b" containerName="cleanup"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235120    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af53d76-a48c-4319-9e29-d983fcdd829f" containerName="main"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235126    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af53d76-a48c-4319-9e29-d983fcdd829f" containerName="main"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235134    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" containerName="main"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235139    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" containerName="main"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235149    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d18851-5efb-4185-bc18-7ba36328e49a" containerName="cleanup"
Jul 27 21:19:56.235188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235154    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d18851-5efb-4185-bc18-7ba36328e49a" containerName="cleanup"
Jul 27 21:19:56.235452 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235211    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b946c3-cd56-484b-b7eb-7427d27aa13b" containerName="cleanup"
Jul 27 21:19:56.235452 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235220    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d18851-5efb-4185-bc18-7ba36328e49a" containerName="cleanup"
Jul 27 21:19:56.235452 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235225    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95dc8b6-8c53-404b-ac0d-b64b7cb524fa" containerName="main"
Jul 27 21:19:56.235452 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.235230    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af53d76-a48c-4319-9e29-d983fcdd829f" containerName="main"
Jul 27 21:19:56.245982 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.245958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.248951 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.248909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"default-dockercfg-6qmtm\""
Jul 27 21:19:56.249831 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.249810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"kube-root-ca.crt\""
Jul 27 21:19:56.250073 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.249810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"openshift-service-ca.crt\""
Jul 27 21:19:56.250188 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.249863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"sub-test-model-ba2511-kserve-self-signed-certs\""
Jul 27 21:19:56.250261 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.250208    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"]
Jul 27 21:19:56.271579 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.271537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-model-cache\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.271680 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.271610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-home\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.271680 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.271657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce82993-79ef-4115-84d4-f0e370a62c64-tls-certs\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.271752 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.271679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gdh\" (UniqueName: \"kubernetes.io/projected/8ce82993-79ef-4115-84d4-f0e370a62c64-kube-api-access-c9gdh\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.271793 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.271775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-tmp-dir\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.271828 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.271818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-dshm\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372213 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-tmp-dir\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372213 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-dshm\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372472 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-model-cache\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372472 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-home\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372472 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce82993-79ef-4115-84d4-f0e370a62c64-tls-certs\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372472 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gdh\" (UniqueName: \"kubernetes.io/projected/8ce82993-79ef-4115-84d4-f0e370a62c64-kube-api-access-c9gdh\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372710 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-tmp-dir\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372710 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-home\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.372781 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.372738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-model-cache\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.374594 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.374570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-dshm\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.374759 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.374743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce82993-79ef-4115-84d4-f0e370a62c64-tls-certs\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.380218 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.380202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gdh\" (UniqueName: \"kubernetes.io/projected/8ce82993-79ef-4115-84d4-f0e370a62c64-kube-api-access-c9gdh\") pod \"sub-test-model-ba2511-kserve-666b4ddb99-rrw59\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") " pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.556704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.556644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:19:56.685440 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.685413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"]
Jul 27 21:19:56.687473 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:19:56.687447    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ce82993_79ef_4115_84d4_f0e370a62c64.slice/crio-21b8f1e516327d76e2cdec85f2820cef30eabe5d665b2f901d088db758b91a2f WatchSource:0}: Error finding container 21b8f1e516327d76e2cdec85f2820cef30eabe5d665b2f901d088db758b91a2f: Status 404 returned error can't find the container with id 21b8f1e516327d76e2cdec85f2820cef30eabe5d665b2f901d088db758b91a2f
Jul 27 21:19:56.763612 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.763581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59" event={"ID":"8ce82993-79ef-4115-84d4-f0e370a62c64","Type":"ContainerStarted","Data":"a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a"}
Jul 27 21:19:56.763762 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:19:56.763625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59" event={"ID":"8ce82993-79ef-4115-84d4-f0e370a62c64","Type":"ContainerStarted","Data":"21b8f1e516327d76e2cdec85f2820cef30eabe5d665b2f901d088db758b91a2f"}
Jul 27 21:20:00.039076 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:00.038967    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59" podStartSLOduration=4.038947304 podStartE2EDuration="4.038947304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:56.782431217 +0000 UTC m=+2690.662579923" watchObservedRunningTime="2026-07-27 21:20:00.038947304 +0000 UTC m=+2693.919096007"
Jul 27 21:20:00.039862 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:00.039845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:20:06.557841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.557805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:20:06.558297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.557858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:20:06.570034 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.570009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:20:06.763250 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.763225    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:20:06.768409 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.768386    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:20:06.823669 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.823578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:20:06.939364 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.939333    2581 scope.go:117] "RemoveContainer" containerID="5d022a83ceaf5ef1f64e0c6472986a323896fb39e71f79634f9f7c16c7be6b36"
Jul 27 21:20:06.947320 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:06.947299    2581 scope.go:117] "RemoveContainer" containerID="5d2be71a4cd02ff73dc734822deeb29427a766c04aba2a41271d1fa7095981ee"
Jul 27 21:20:28.637139 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:28.637104    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:20:48.237364 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:48.237330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:20:54.717716 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:20:54.717681    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"]
Jul 27 21:21:03.039151 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.039118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"]
Jul 27 21:21:03.042845 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.042825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.046196 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.046166    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"openshift-service-ca.crt\""
Jul 27 21:21:03.046317 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.046209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"rate-test-model-69909d-kserve-self-signed-certs\""
Jul 27 21:21:03.046317 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.046229    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"default-dockercfg-lwsxj\""
Jul 27 21:21:03.046317 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.046238    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-69909d\"/\"kube-root-ca.crt\""
Jul 27 21:21:03.051423 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.051402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"]
Jul 27 21:21:03.158341 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.158305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-model-cache\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.158510 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.158347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8cll\" (UniqueName: \"kubernetes.io/projected/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-kube-api-access-g8cll\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.158510 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.158439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-home\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.158510 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.158474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tmp-dir\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.158510 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.158506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tls-certs\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.158695 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.158601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-dshm\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.259988 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.259942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-dshm\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.259988 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-model-cache\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260231 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8cll\" (UniqueName: \"kubernetes.io/projected/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-kube-api-access-g8cll\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260231 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-home\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260231 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tmp-dir\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260231 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tls-certs\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260472 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-model-cache\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260544 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tmp-dir\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.260640 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.260588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-home\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.262375 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.262358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-dshm\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.262538 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.262520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tls-certs\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.267959 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.267938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8cll\" (UniqueName: \"kubernetes.io/projected/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-kube-api-access-g8cll\") pod \"rate-test-model-69909d-kserve-844645fd9f-bhbbh\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") " pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.352440 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.352409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:03.483497 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:03.483473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"]
Jul 27 21:21:03.485313 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:21:03.485286    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aa24bb4_d7e6_4aea_8ad0_c5b828164617.slice/crio-86bc6aab866181f5f11b2d2d6e1143d87907417dd1140b46910328f3e1268c20 WatchSource:0}: Error finding container 86bc6aab866181f5f11b2d2d6e1143d87907417dd1140b46910328f3e1268c20: Status 404 returned error can't find the container with id 86bc6aab866181f5f11b2d2d6e1143d87907417dd1140b46910328f3e1268c20
Jul 27 21:21:04.081566 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:04.081507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh" event={"ID":"6aa24bb4-d7e6-4aea-8ad0-c5b828164617","Type":"ContainerStarted","Data":"93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77"}
Jul 27 21:21:04.081566 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:04.081569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh" event={"ID":"6aa24bb4-d7e6-4aea-8ad0-c5b828164617","Type":"ContainerStarted","Data":"86bc6aab866181f5f11b2d2d6e1143d87907417dd1140b46910328f3e1268c20"}
Jul 27 21:21:04.100050 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:04.099998    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh" podStartSLOduration=1.099981994 podStartE2EDuration="1.099981994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:21:04.098599369 +0000 UTC m=+2757.978748070" watchObservedRunningTime="2026-07-27 21:21:04.099981994 +0000 UTC m=+2757.980130697"
Jul 27 21:21:05.934908 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:05.934870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:21:09.737674 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:09.737604    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59" podUID="8ce82993-79ef-4115-84d4-f0e370a62c64" containerName="main" containerID="cri-o://a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a" gracePeriod=45
Jul 27 21:21:09.987777 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:09.987714    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:21:10.109287 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.109256    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ce82993-79ef-4115-84d4-f0e370a62c64" containerID="a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a" exitCode=0
Jul 27 21:21:10.109456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.109305    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"
Jul 27 21:21:10.109456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.109331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59" event={"ID":"8ce82993-79ef-4115-84d4-f0e370a62c64","Type":"ContainerDied","Data":"a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a"}
Jul 27 21:21:10.109456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.109376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59" event={"ID":"8ce82993-79ef-4115-84d4-f0e370a62c64","Type":"ContainerDied","Data":"21b8f1e516327d76e2cdec85f2820cef30eabe5d665b2f901d088db758b91a2f"}
Jul 27 21:21:10.109456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.109398    2581 scope.go:117] "RemoveContainer" containerID="a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a"
Jul 27 21:21:10.117744 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.117725    2581 scope.go:117] "RemoveContainer" containerID="a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a"
Jul 27 21:21:10.117996 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:21:10.117977    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a\": container with ID starting with a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a not found: ID does not exist" containerID="a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a"
Jul 27 21:21:10.118075 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.118004    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a"} err="failed to get container status \"a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a\": rpc error: code = NotFound desc = could not find container \"a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a\": container with ID starting with a2a4213e74a018868749cb1c1b6667ae2cd8affd5059337eaa652545f402a92a not found: ID does not exist"
Jul 27 21:21:10.119235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-dshm\") pod \"8ce82993-79ef-4115-84d4-f0e370a62c64\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") "
Jul 27 21:21:10.119306 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119247    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-tmp-dir\") pod \"8ce82993-79ef-4115-84d4-f0e370a62c64\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") "
Jul 27 21:21:10.119306 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9gdh\" (UniqueName: \"kubernetes.io/projected/8ce82993-79ef-4115-84d4-f0e370a62c64-kube-api-access-c9gdh\") pod \"8ce82993-79ef-4115-84d4-f0e370a62c64\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") "
Jul 27 21:21:10.119424 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-model-cache\") pod \"8ce82993-79ef-4115-84d4-f0e370a62c64\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") "
Jul 27 21:21:10.119424 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119389    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce82993-79ef-4115-84d4-f0e370a62c64-tls-certs\") pod \"8ce82993-79ef-4115-84d4-f0e370a62c64\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") "
Jul 27 21:21:10.119532 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119422    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-home\") pod \"8ce82993-79ef-4115-84d4-f0e370a62c64\" (UID: \"8ce82993-79ef-4115-84d4-f0e370a62c64\") "
Jul 27 21:21:10.119532 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119485    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ce82993-79ef-4115-84d4-f0e370a62c64" (UID: "8ce82993-79ef-4115-84d4-f0e370a62c64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:10.119649 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119615    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ce82993-79ef-4115-84d4-f0e370a62c64" (UID: "8ce82993-79ef-4115-84d4-f0e370a62c64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:10.119740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-home" (OuterVolumeSpecName: "home") pod "8ce82993-79ef-4115-84d4-f0e370a62c64" (UID: "8ce82993-79ef-4115-84d4-f0e370a62c64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:10.119740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119728    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:21:10.119885 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.119747    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:21:10.121269 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.121248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce82993-79ef-4115-84d4-f0e370a62c64-kube-api-access-c9gdh" (OuterVolumeSpecName: "kube-api-access-c9gdh") pod "8ce82993-79ef-4115-84d4-f0e370a62c64" (UID: "8ce82993-79ef-4115-84d4-f0e370a62c64"). InnerVolumeSpecName "kube-api-access-c9gdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:21:10.121366 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.121351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-dshm" (OuterVolumeSpecName: "dshm") pod "8ce82993-79ef-4115-84d4-f0e370a62c64" (UID: "8ce82993-79ef-4115-84d4-f0e370a62c64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:10.121424 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.121402    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce82993-79ef-4115-84d4-f0e370a62c64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ce82993-79ef-4115-84d4-f0e370a62c64" (UID: "8ce82993-79ef-4115-84d4-f0e370a62c64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:21:10.220337 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.220301    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce82993-79ef-4115-84d4-f0e370a62c64-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:21:10.220337 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.220328    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:21:10.220337 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.220339    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce82993-79ef-4115-84d4-f0e370a62c64-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:21:10.220585 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.220352    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gdh\" (UniqueName: \"kubernetes.io/projected/8ce82993-79ef-4115-84d4-f0e370a62c64-kube-api-access-c9gdh\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:21:10.430333 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.430292    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"]
Jul 27 21:21:10.436987 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.436964    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/sub-test-model-ba2511-kserve-666b4ddb99-rrw59"]
Jul 27 21:21:10.734722 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:10.734642    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce82993-79ef-4115-84d4-f0e370a62c64" path="/var/lib/kubelet/pods/8ce82993-79ef-4115-84d4-f0e370a62c64/volumes"
Jul 27 21:21:13.352848 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:13.352815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:13.355141 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:13.352867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:13.365045 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:13.365027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:14.139799 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:14.139774    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:21:27.130811 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.130771    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"]
Jul 27 21:21:27.131299 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.131132    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce82993-79ef-4115-84d4-f0e370a62c64" containerName="main"
Jul 27 21:21:27.131299 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.131146    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce82993-79ef-4115-84d4-f0e370a62c64" containerName="main"
Jul 27 21:21:27.131299 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.131209    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce82993-79ef-4115-84d4-f0e370a62c64" containerName="main"
Jul 27 21:21:27.134331 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.134314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.136963 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.136943    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"default-dockercfg-6qmtm\""
Jul 27 21:21:27.137802 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.137782    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"openshift-service-ca.crt\""
Jul 27 21:21:27.137901 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.137807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"kube-root-ca.crt\""
Jul 27 21:21:27.137961 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.137902    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba2511\"/\"rate-test-model-ba2511-kserve-self-signed-certs\""
Jul 27 21:21:27.144921 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.144901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"]
Jul 27 21:21:27.172811 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.172789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-dshm\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.172921 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.172820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-tmp-dir\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.172921 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.172850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-model-cache\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.172921 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.172865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d906ed-b940-4389-bdec-d7060079d4b4-tls-certs\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.173023 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.172937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxgz\" (UniqueName: \"kubernetes.io/projected/70d906ed-b940-4389-bdec-d7060079d4b4-kube-api-access-plxgz\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.173023 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.172967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-home\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.273565 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.273512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxgz\" (UniqueName: \"kubernetes.io/projected/70d906ed-b940-4389-bdec-d7060079d4b4-kube-api-access-plxgz\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.273724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.273578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-home\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.273724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.273599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-dshm\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.273724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.273619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-tmp-dir\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.273724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.273649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-model-cache\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.273724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.273674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d906ed-b940-4389-bdec-d7060079d4b4-tls-certs\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.274060 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.274039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-tmp-dir\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.274130 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.274088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-home\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.274130 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.274120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-model-cache\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.275761 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.275744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-dshm\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.276070 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.276054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d906ed-b940-4389-bdec-d7060079d4b4-tls-certs\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.282684 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.282659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxgz\" (UniqueName: \"kubernetes.io/projected/70d906ed-b940-4389-bdec-d7060079d4b4-kube-api-access-plxgz\") pod \"rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") " pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.443152 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.443060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:27.574298 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:27.574273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"]
Jul 27 21:21:27.576856 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:21:27.576825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70d906ed_b940_4389_bdec_d7060079d4b4.slice/crio-296d92d5723f9267f148aa229adea010a1507a61e511e4b5b41222d45ea653d3 WatchSource:0}: Error finding container 296d92d5723f9267f148aa229adea010a1507a61e511e4b5b41222d45ea653d3: Status 404 returned error can't find the container with id 296d92d5723f9267f148aa229adea010a1507a61e511e4b5b41222d45ea653d3
Jul 27 21:21:28.193800 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:28.193762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2" event={"ID":"70d906ed-b940-4389-bdec-d7060079d4b4","Type":"ContainerStarted","Data":"cb55d4e0f9179b65f09aa8f03a6dd3eb324e223ee0ddb3b4b14e27e3ab4579b3"}
Jul 27 21:21:28.193800 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:28.193805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2" event={"ID":"70d906ed-b940-4389-bdec-d7060079d4b4","Type":"ContainerStarted","Data":"296d92d5723f9267f148aa229adea010a1507a61e511e4b5b41222d45ea653d3"}
Jul 27 21:21:28.209080 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:28.209034    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2" podStartSLOduration=1.209019291 podStartE2EDuration="1.209019291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:21:28.20792095 +0000 UTC m=+2782.088069654" watchObservedRunningTime="2026-07-27 21:21:28.209019291 +0000 UTC m=+2782.089168002"
Jul 27 21:21:30.926516 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:30.926475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:21:37.443951 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:37.443910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:37.444406 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:37.443970    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:37.456296 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:37.456267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:21:38.250774 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:21:38.250745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:22:03.330533 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:03.330496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:22:17.534720 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:17.534689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:22:31.731298 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:31.731263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:22:36.546881 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:36.546849    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"]
Jul 27 21:22:37.353984 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:37.353951    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"]
Jul 27 21:22:38.373438 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.373403    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-139641640635696"]
Jul 27 21:22:38.377000 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.376984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:38.379211 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.379183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p9zrj\""
Jul 27 21:22:38.382146 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.382123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641640635696"]
Jul 27 21:22:38.504080 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.504042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824fd\" (UniqueName: \"kubernetes.io/projected/21b3624b-a6bb-49dc-a4c7-dbae47b4576e-kube-api-access-824fd\") pod \"test-curl-7706-139641640635696\" (UID: \"21b3624b-a6bb-49dc-a4c7-dbae47b4576e\") " pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:38.605582 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.605528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824fd\" (UniqueName: \"kubernetes.io/projected/21b3624b-a6bb-49dc-a4c7-dbae47b4576e-kube-api-access-824fd\") pod \"test-curl-7706-139641640635696\" (UID: \"21b3624b-a6bb-49dc-a4c7-dbae47b4576e\") " pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:38.613773 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.613741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824fd\" (UniqueName: \"kubernetes.io/projected/21b3624b-a6bb-49dc-a4c7-dbae47b4576e-kube-api-access-824fd\") pod \"test-curl-7706-139641640635696\" (UID: \"21b3624b-a6bb-49dc-a4c7-dbae47b4576e\") " pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:38.686964 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.686891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:38.809740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.809711    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641640635696"]
Jul 27 21:22:38.811613 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:22:38.811588    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b3624b_a6bb_49dc_a4c7_dbae47b4576e.slice/crio-d83926f1728cd805802b6dd4e155ce4129134bccbc7ff4c0e04d178b780f0c4c WatchSource:0}: Error finding container d83926f1728cd805802b6dd4e155ce4129134bccbc7ff4c0e04d178b780f0c4c: Status 404 returned error can't find the container with id d83926f1728cd805802b6dd4e155ce4129134bccbc7ff4c0e04d178b780f0c4c
Jul 27 21:22:38.812808 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:38.812789    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:22:39.529486 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:39.529460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640635696" event={"ID":"21b3624b-a6bb-49dc-a4c7-dbae47b4576e","Type":"ContainerStarted","Data":"d83926f1728cd805802b6dd4e155ce4129134bccbc7ff4c0e04d178b780f0c4c"}
Jul 27 21:22:40.535339 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:40.535298    2581 generic.go:356] "Generic (PLEG): container finished" podID="21b3624b-a6bb-49dc-a4c7-dbae47b4576e" containerID="545a067efaca793ec44e523b9c70fed6605e3912282cee65cde8b49c03a87686" exitCode=0
Jul 27 21:22:40.535749 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:40.535385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640635696" event={"ID":"21b3624b-a6bb-49dc-a4c7-dbae47b4576e","Type":"ContainerDied","Data":"545a067efaca793ec44e523b9c70fed6605e3912282cee65cde8b49c03a87686"}
Jul 27 21:22:41.665184 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.665158    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:41.698765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.698744    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7706-139641640635696_21b3624b-a6bb-49dc-a4c7-dbae47b4576e/test-curl-7706-139641640635696/0.log"
Jul 27 21:22:41.719082 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.719041    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-139641640635696"]
Jul 27 21:22:41.720970 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.720949    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-139641640635696"]
Jul 27 21:22:41.733399 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.733384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-824fd\" (UniqueName: \"kubernetes.io/projected/21b3624b-a6bb-49dc-a4c7-dbae47b4576e-kube-api-access-824fd\") pod \"21b3624b-a6bb-49dc-a4c7-dbae47b4576e\" (UID: \"21b3624b-a6bb-49dc-a4c7-dbae47b4576e\") "
Jul 27 21:22:41.735445 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.735426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b3624b-a6bb-49dc-a4c7-dbae47b4576e-kube-api-access-824fd" (OuterVolumeSpecName: "kube-api-access-824fd") pod "21b3624b-a6bb-49dc-a4c7-dbae47b4576e" (UID: "21b3624b-a6bb-49dc-a4c7-dbae47b4576e"). InnerVolumeSpecName "kube-api-access-824fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:41.834956 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.834882    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824fd\" (UniqueName: \"kubernetes.io/projected/21b3624b-a6bb-49dc-a4c7-dbae47b4576e-kube-api-access-824fd\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:41.839161 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.839135    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-139641640632368"]
Jul 27 21:22:41.839509 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.839497    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b3624b-a6bb-49dc-a4c7-dbae47b4576e" containerName="test-curl-7706-139641640635696"
Jul 27 21:22:41.839573 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.839512    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b3624b-a6bb-49dc-a4c7-dbae47b4576e" containerName="test-curl-7706-139641640635696"
Jul 27 21:22:41.839613 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.839588    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b3624b-a6bb-49dc-a4c7-dbae47b4576e" containerName="test-curl-7706-139641640635696"
Jul 27 21:22:41.842297 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.842281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:41.846120 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.846094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641640632368"]
Jul 27 21:22:41.936023 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:41.935989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckh8w\" (UniqueName: \"kubernetes.io/projected/15da8150-376d-4a4c-982c-8d5b375ce4c7-kube-api-access-ckh8w\") pod \"test-curl-7706-139641640632368\" (UID: \"15da8150-376d-4a4c-982c-8d5b375ce4c7\") " pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:42.036883 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.036848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh8w\" (UniqueName: \"kubernetes.io/projected/15da8150-376d-4a4c-982c-8d5b375ce4c7-kube-api-access-ckh8w\") pod \"test-curl-7706-139641640632368\" (UID: \"15da8150-376d-4a4c-982c-8d5b375ce4c7\") " pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:42.045234 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.045207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckh8w\" (UniqueName: \"kubernetes.io/projected/15da8150-376d-4a4c-982c-8d5b375ce4c7-kube-api-access-ckh8w\") pod \"test-curl-7706-139641640632368\" (UID: \"15da8150-376d-4a4c-982c-8d5b375ce4c7\") " pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:42.152887 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.152850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:42.277006 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.276985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641640632368"]
Jul 27 21:22:42.279094 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:22:42.279070    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15da8150_376d_4a4c_982c_8d5b375ce4c7.slice/crio-d64a481604c376287a311c5602a4da3ceb5609d2a6e794fc70547cbf5fe339c8 WatchSource:0}: Error finding container d64a481604c376287a311c5602a4da3ceb5609d2a6e794fc70547cbf5fe339c8: Status 404 returned error can't find the container with id d64a481604c376287a311c5602a4da3ceb5609d2a6e794fc70547cbf5fe339c8
Jul 27 21:22:42.547028 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.546926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640632368" event={"ID":"15da8150-376d-4a4c-982c-8d5b375ce4c7","Type":"ContainerStarted","Data":"d64a481604c376287a311c5602a4da3ceb5609d2a6e794fc70547cbf5fe339c8"}
Jul 27 21:22:42.548179 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.548160    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d83926f1728cd805802b6dd4e155ce4129134bccbc7ff4c0e04d178b780f0c4c"
Jul 27 21:22:42.548294 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.548190    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:42.558446 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:42.558421    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-139641640635696\" is forbidden: User \"system:node:ip-10-0-142-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-163.ec2.internal' and this object" podUID="21b3624b-a6bb-49dc-a4c7-dbae47b4576e" pod="opendatahub/test-curl-7706-139641640635696"
Jul 27 21:22:42.733982 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:42.733959    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b3624b-a6bb-49dc-a4c7-dbae47b4576e" path="/var/lib/kubelet/pods/21b3624b-a6bb-49dc-a4c7-dbae47b4576e/volumes"
Jul 27 21:22:43.553820 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:43.553787    2581 generic.go:356] "Generic (PLEG): container finished" podID="15da8150-376d-4a4c-982c-8d5b375ce4c7" containerID="c8c5b9b0a6bc7951302aa0848dcb87279a496fc41a1aa3925be016029c4da161" exitCode=0
Jul 27 21:22:43.554013 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:43.553881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640632368" event={"ID":"15da8150-376d-4a4c-982c-8d5b375ce4c7","Type":"ContainerDied","Data":"c8c5b9b0a6bc7951302aa0848dcb87279a496fc41a1aa3925be016029c4da161"}
Jul 27 21:22:44.692223 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.692193    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:44.711770 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.711746    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7706-139641640632368_15da8150-376d-4a4c-982c-8d5b375ce4c7/test-curl-7706-139641640632368/0.log"
Jul 27 21:22:44.737982 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.737956    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-139641640632368"]
Jul 27 21:22:44.742670 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.742648    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-139641640632368"]
Jul 27 21:22:44.760467 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.760435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckh8w\" (UniqueName: \"kubernetes.io/projected/15da8150-376d-4a4c-982c-8d5b375ce4c7-kube-api-access-ckh8w\") pod \"15da8150-376d-4a4c-982c-8d5b375ce4c7\" (UID: \"15da8150-376d-4a4c-982c-8d5b375ce4c7\") "
Jul 27 21:22:44.762226 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.762203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15da8150-376d-4a4c-982c-8d5b375ce4c7-kube-api-access-ckh8w" (OuterVolumeSpecName: "kube-api-access-ckh8w") pod "15da8150-376d-4a4c-982c-8d5b375ce4c7" (UID: "15da8150-376d-4a4c-982c-8d5b375ce4c7"). InnerVolumeSpecName "kube-api-access-ckh8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:44.861947 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.861918    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckh8w\" (UniqueName: \"kubernetes.io/projected/15da8150-376d-4a4c-982c-8d5b375ce4c7-kube-api-access-ckh8w\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:44.911045 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.911018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-139641636775808"]
Jul 27 21:22:44.911372 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.911360    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" containerName="test-curl-7706-139641640632368"
Jul 27 21:22:44.911416 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.911374    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" containerName="test-curl-7706-139641640632368"
Jul 27 21:22:44.911453 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.911442    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" containerName="test-curl-7706-139641640632368"
Jul 27 21:22:44.913498 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.913484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:44.918670 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:44.918647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641636775808"]
Jul 27 21:22:45.064038 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.063999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm7cf\" (UniqueName: \"kubernetes.io/projected/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22-kube-api-access-fm7cf\") pod \"test-curl-7706-139641636775808\" (UID: \"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22\") " pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:45.164856 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.164767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm7cf\" (UniqueName: \"kubernetes.io/projected/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22-kube-api-access-fm7cf\") pod \"test-curl-7706-139641636775808\" (UID: \"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22\") " pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:45.173286 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.173260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm7cf\" (UniqueName: \"kubernetes.io/projected/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22-kube-api-access-fm7cf\") pod \"test-curl-7706-139641636775808\" (UID: \"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22\") " pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:45.223449 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.223421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:45.344699 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.344673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641636775808"]
Jul 27 21:22:45.346415 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:22:45.346383    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f4c7c25_9fc1_4690_90a1_8e0f5ac22c22.slice/crio-7820263fd39626471626d194707949964571ce9768235ab7f5db3a110d455a9a WatchSource:0}: Error finding container 7820263fd39626471626d194707949964571ce9768235ab7f5db3a110d455a9a: Status 404 returned error can't find the container with id 7820263fd39626471626d194707949964571ce9768235ab7f5db3a110d455a9a
Jul 27 21:22:45.564355 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.564270    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:45.564355 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.564313    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64a481604c376287a311c5602a4da3ceb5609d2a6e794fc70547cbf5fe339c8"
Jul 27 21:22:45.565446 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:45.565423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641636775808" event={"ID":"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22","Type":"ContainerStarted","Data":"7820263fd39626471626d194707949964571ce9768235ab7f5db3a110d455a9a"}
Jul 27 21:22:45.575079 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:45.575054    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-139641640632368\" is forbidden: User \"system:node:ip-10-0-142-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-163.ec2.internal' and this object" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:46.571656 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:46.571623    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" containerID="aabc646b36d722a7ad8fce6532ed9a87c023a72677b95093977d5042594b52fe" exitCode=0
Jul 27 21:22:46.572019 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:46.571673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641636775808" event={"ID":"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22","Type":"ContainerDied","Data":"aabc646b36d722a7ad8fce6532ed9a87c023a72677b95093977d5042594b52fe"}
Jul 27 21:22:46.574233 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:46.574211    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-139641640632368\" is forbidden: User \"system:node:ip-10-0-142-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-163.ec2.internal' and this object" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:46.733005 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:46.732970    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-139641640632368\" is forbidden: User \"system:node:ip-10-0-142-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-163.ec2.internal' and this object" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" pod="opendatahub/test-curl-7706-139641640632368"
Jul 27 21:22:46.734099 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:46.734080    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15da8150-376d-4a4c-982c-8d5b375ce4c7" path="/var/lib/kubelet/pods/15da8150-376d-4a4c-982c-8d5b375ce4c7/volumes"
Jul 27 21:22:47.702804 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.702783    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:47.725214 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.725194    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7706-139641636775808_7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22/test-curl-7706-139641636775808/0.log"
Jul 27 21:22:47.747959 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.747935    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-139641636775808"]
Jul 27 21:22:47.754120 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.754098    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-139641636775808"]
Jul 27 21:22:47.789563 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.789526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm7cf\" (UniqueName: \"kubernetes.io/projected/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22-kube-api-access-fm7cf\") pod \"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22\" (UID: \"7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22\") "
Jul 27 21:22:47.791454 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.791430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22-kube-api-access-fm7cf" (OuterVolumeSpecName: "kube-api-access-fm7cf") pod "7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" (UID: "7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22"). InnerVolumeSpecName "kube-api-access-fm7cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:47.891398 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:47.891371    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm7cf\" (UniqueName: \"kubernetes.io/projected/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22-kube-api-access-fm7cf\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:48.335256 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:48.335179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:22:48.586083 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:48.586004    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7820263fd39626471626d194707949964571ce9768235ab7f5db3a110d455a9a"
Jul 27 21:22:48.586083 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:48.586012    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:48.595219 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:48.595188    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-139641636775808\" is forbidden: User \"system:node:ip-10-0-142-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-163.ec2.internal' and this object" podUID="7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" pod="opendatahub/test-curl-7706-139641636775808"
Jul 27 21:22:48.734084 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:48.734051    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" path="/var/lib/kubelet/pods/7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22/volumes"
Jul 27 21:22:51.565241 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.565192    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh" podUID="6aa24bb4-d7e6-4aea-8ad0-c5b828164617" containerName="main" containerID="cri-o://93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77" gracePeriod=45
Jul 27 21:22:51.808661 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.808638    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:22:51.926721 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-home\") pod \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") "
Jul 27 21:22:51.926886 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tmp-dir\") pod \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") "
Jul 27 21:22:51.926886 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926773    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tls-certs\") pod \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") "
Jul 27 21:22:51.926886 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-dshm\") pod \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") "
Jul 27 21:22:51.926886 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926841    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-model-cache\") pod \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") "
Jul 27 21:22:51.927127 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926938    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8cll\" (UniqueName: \"kubernetes.io/projected/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-kube-api-access-g8cll\") pod \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\" (UID: \"6aa24bb4-d7e6-4aea-8ad0-c5b828164617\") "
Jul 27 21:22:51.927127 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.926979    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-home" (OuterVolumeSpecName: "home") pod "6aa24bb4-d7e6-4aea-8ad0-c5b828164617" (UID: "6aa24bb4-d7e6-4aea-8ad0-c5b828164617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:51.927127 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.927094    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6aa24bb4-d7e6-4aea-8ad0-c5b828164617" (UID: "6aa24bb4-d7e6-4aea-8ad0-c5b828164617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:51.927267 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.927131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-model-cache" (OuterVolumeSpecName: "model-cache") pod "6aa24bb4-d7e6-4aea-8ad0-c5b828164617" (UID: "6aa24bb4-d7e6-4aea-8ad0-c5b828164617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:51.927267 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.927243    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:51.927267 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.927260    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:51.927368 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.927269    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:51.928991 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.928967    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-dshm" (OuterVolumeSpecName: "dshm") pod "6aa24bb4-d7e6-4aea-8ad0-c5b828164617" (UID: "6aa24bb4-d7e6-4aea-8ad0-c5b828164617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:51.929102 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.929011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6aa24bb4-d7e6-4aea-8ad0-c5b828164617" (UID: "6aa24bb4-d7e6-4aea-8ad0-c5b828164617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:22:51.929102 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:51.929021    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-kube-api-access-g8cll" (OuterVolumeSpecName: "kube-api-access-g8cll") pod "6aa24bb4-d7e6-4aea-8ad0-c5b828164617" (UID: "6aa24bb4-d7e6-4aea-8ad0-c5b828164617"). InnerVolumeSpecName "kube-api-access-g8cll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:52.027849 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.027809    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.027849 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.027840    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.027849 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.027852    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8cll\" (UniqueName: \"kubernetes.io/projected/6aa24bb4-d7e6-4aea-8ad0-c5b828164617-kube-api-access-g8cll\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.371910 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.371871    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2" podUID="70d906ed-b940-4389-bdec-d7060079d4b4" containerName="main" containerID="cri-o://cb55d4e0f9179b65f09aa8f03a6dd3eb324e223ee0ddb3b4b14e27e3ab4579b3" gracePeriod=45
Jul 27 21:22:52.607225 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.607195    2581 generic.go:356] "Generic (PLEG): container finished" podID="70d906ed-b940-4389-bdec-d7060079d4b4" containerID="cb55d4e0f9179b65f09aa8f03a6dd3eb324e223ee0ddb3b4b14e27e3ab4579b3" exitCode=0
Jul 27 21:22:52.607616 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.607276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2" event={"ID":"70d906ed-b940-4389-bdec-d7060079d4b4","Type":"ContainerDied","Data":"cb55d4e0f9179b65f09aa8f03a6dd3eb324e223ee0ddb3b4b14e27e3ab4579b3"}
Jul 27 21:22:52.607616 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.607326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2" event={"ID":"70d906ed-b940-4389-bdec-d7060079d4b4","Type":"ContainerDied","Data":"296d92d5723f9267f148aa229adea010a1507a61e511e4b5b41222d45ea653d3"}
Jul 27 21:22:52.607616 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.607349    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296d92d5723f9267f148aa229adea010a1507a61e511e4b5b41222d45ea653d3"
Jul 27 21:22:52.608683 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.608662    2581 generic.go:356] "Generic (PLEG): container finished" podID="6aa24bb4-d7e6-4aea-8ad0-c5b828164617" containerID="93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77" exitCode=0
Jul 27 21:22:52.608786 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.608705    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"
Jul 27 21:22:52.608786 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.608718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh" event={"ID":"6aa24bb4-d7e6-4aea-8ad0-c5b828164617","Type":"ContainerDied","Data":"93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77"}
Jul 27 21:22:52.608786 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.608753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh" event={"ID":"6aa24bb4-d7e6-4aea-8ad0-c5b828164617","Type":"ContainerDied","Data":"86bc6aab866181f5f11b2d2d6e1143d87907417dd1140b46910328f3e1268c20"}
Jul 27 21:22:52.608786 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.608774    2581 scope.go:117] "RemoveContainer" containerID="93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77"
Jul 27 21:22:52.619653 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.619633    2581 scope.go:117] "RemoveContainer" containerID="93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77"
Jul 27 21:22:52.619905 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:52.619885    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77\": container with ID starting with 93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77 not found: ID does not exist" containerID="93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77"
Jul 27 21:22:52.619972 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.619914    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77"} err="failed to get container status \"93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77\": rpc error: code = NotFound desc = could not find container \"93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77\": container with ID starting with 93cc6072b49d31c538aeb98b8c507a21c2f9f5bd7b86c29135dc27b8ed24fa77 not found: ID does not exist"
Jul 27 21:22:52.624170 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.624128    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:22:52.634683 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.634664    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"]
Jul 27 21:22:52.639668 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.639651    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-69909d/rate-test-model-69909d-kserve-844645fd9f-bhbbh"]
Jul 27 21:22:52.733891 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.733862    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa24bb4-d7e6-4aea-8ad0-c5b828164617" path="/var/lib/kubelet/pods/6aa24bb4-d7e6-4aea-8ad0-c5b828164617/volumes"
Jul 27 21:22:52.734410 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d906ed-b940-4389-bdec-d7060079d4b4-tls-certs\") pod \"70d906ed-b940-4389-bdec-d7060079d4b4\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") "
Jul 27 21:22:52.734501 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-tmp-dir\") pod \"70d906ed-b940-4389-bdec-d7060079d4b4\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") "
Jul 27 21:22:52.734501 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-home\") pod \"70d906ed-b940-4389-bdec-d7060079d4b4\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") "
Jul 27 21:22:52.734501 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-dshm\") pod \"70d906ed-b940-4389-bdec-d7060079d4b4\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") "
Jul 27 21:22:52.734672 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-model-cache\") pod \"70d906ed-b940-4389-bdec-d7060079d4b4\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") "
Jul 27 21:22:52.734672 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plxgz\" (UniqueName: \"kubernetes.io/projected/70d906ed-b940-4389-bdec-d7060079d4b4-kube-api-access-plxgz\") pod \"70d906ed-b940-4389-bdec-d7060079d4b4\" (UID: \"70d906ed-b940-4389-bdec-d7060079d4b4\") "
Jul 27 21:22:52.734774 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-home" (OuterVolumeSpecName: "home") pod "70d906ed-b940-4389-bdec-d7060079d4b4" (UID: "70d906ed-b940-4389-bdec-d7060079d4b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:52.734774 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70d906ed-b940-4389-bdec-d7060079d4b4" (UID: "70d906ed-b940-4389-bdec-d7060079d4b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:52.734915 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734890    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.734915 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "70d906ed-b940-4389-bdec-d7060079d4b4" (UID: "70d906ed-b940-4389-bdec-d7060079d4b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:52.734915 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.734916    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.736629 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.736599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-dshm" (OuterVolumeSpecName: "dshm") pod "70d906ed-b940-4389-bdec-d7060079d4b4" (UID: "70d906ed-b940-4389-bdec-d7060079d4b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:52.736717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.736653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d906ed-b940-4389-bdec-d7060079d4b4-kube-api-access-plxgz" (OuterVolumeSpecName: "kube-api-access-plxgz") pod "70d906ed-b940-4389-bdec-d7060079d4b4" (UID: "70d906ed-b940-4389-bdec-d7060079d4b4"). InnerVolumeSpecName "kube-api-access-plxgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:52.736717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.736682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d906ed-b940-4389-bdec-d7060079d4b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70d906ed-b940-4389-bdec-d7060079d4b4" (UID: "70d906ed-b940-4389-bdec-d7060079d4b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:22:52.836032 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.835992    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d906ed-b940-4389-bdec-d7060079d4b4-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.836032 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.836026    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.836032 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.836035    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d906ed-b940-4389-bdec-d7060079d4b4-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:52.836271 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:52.836045    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plxgz\" (UniqueName: \"kubernetes.io/projected/70d906ed-b940-4389-bdec-d7060079d4b4-kube-api-access-plxgz\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:53.614442 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.614412    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"
Jul 27 21:22:53.634427 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.634405    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"]
Jul 27 21:22:53.639891 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.639870    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ba2511/rate-test-model-ba2511-kserve-5f9fcbcbb7-8n9r2"]
Jul 27 21:22:53.790761 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.790729    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-139641640572512"]
Jul 27 21:22:53.791085 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791072    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa24bb4-d7e6-4aea-8ad0-c5b828164617" containerName="main"
Jul 27 21:22:53.791131 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791087    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa24bb4-d7e6-4aea-8ad0-c5b828164617" containerName="main"
Jul 27 21:22:53.791131 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791099    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" containerName="test-curl-7706-139641636775808"
Jul 27 21:22:53.791131 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" containerName="test-curl-7706-139641636775808"
Jul 27 21:22:53.791131 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791118    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d906ed-b940-4389-bdec-d7060079d4b4" containerName="main"
Jul 27 21:22:53.791131 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791123    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d906ed-b940-4389-bdec-d7060079d4b4" containerName="main"
Jul 27 21:22:53.791281 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791180    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="70d906ed-b940-4389-bdec-d7060079d4b4" containerName="main"
Jul 27 21:22:53.791281 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791188    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa24bb4-d7e6-4aea-8ad0-c5b828164617" containerName="main"
Jul 27 21:22:53.791281 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.791197    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f4c7c25-9fc1-4690-90a1-8e0f5ac22c22" containerName="test-curl-7706-139641636775808"
Jul 27 21:22:53.794350 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.794333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:53.796567 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.796538    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p9zrj\""
Jul 27 21:22:53.800593 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.800572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641640572512"]
Jul 27 21:22:53.945162 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:53.945077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jv5k\" (UniqueName: \"kubernetes.io/projected/44170b78-193d-43a8-b596-1cf2ebce669f-kube-api-access-6jv5k\") pod \"test-curl-7706-139641640572512\" (UID: \"44170b78-193d-43a8-b596-1cf2ebce669f\") " pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:54.045728 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.045679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jv5k\" (UniqueName: \"kubernetes.io/projected/44170b78-193d-43a8-b596-1cf2ebce669f-kube-api-access-6jv5k\") pod \"test-curl-7706-139641640572512\" (UID: \"44170b78-193d-43a8-b596-1cf2ebce669f\") " pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:54.054486 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.054458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jv5k\" (UniqueName: \"kubernetes.io/projected/44170b78-193d-43a8-b596-1cf2ebce669f-kube-api-access-6jv5k\") pod \"test-curl-7706-139641640572512\" (UID: \"44170b78-193d-43a8-b596-1cf2ebce669f\") " pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:54.104073 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.104040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:54.250113 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.250074    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-139641640572512"]
Jul 27 21:22:54.252341 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:22:54.252310    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44170b78_193d_43a8_b596_1cf2ebce669f.slice/crio-dc5da3d171aee91e8776e0be1af1d630ec8ae6620d238bb4ba37daea5ee6b8d5 WatchSource:0}: Error finding container dc5da3d171aee91e8776e0be1af1d630ec8ae6620d238bb4ba37daea5ee6b8d5: Status 404 returned error can't find the container with id dc5da3d171aee91e8776e0be1af1d630ec8ae6620d238bb4ba37daea5ee6b8d5
Jul 27 21:22:54.621389 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.621357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640572512" event={"ID":"44170b78-193d-43a8-b596-1cf2ebce669f","Type":"ContainerStarted","Data":"a875c1fc4a8c97b536d5e562526f2a9dc120c5d62e86d27a9f364641f3541580"}
Jul 27 21:22:54.621834 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.621777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640572512" event={"ID":"44170b78-193d-43a8-b596-1cf2ebce669f","Type":"ContainerStarted","Data":"dc5da3d171aee91e8776e0be1af1d630ec8ae6620d238bb4ba37daea5ee6b8d5"}
Jul 27 21:22:54.636490 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.636449    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7706-139641640572512" podStartSLOduration=1.3581244909999999 podStartE2EDuration="1.636433819s" totalImagesPullingTime="278.309328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:22:54.63616792 +0000 UTC m=+2868.516316623" watchObservedRunningTime="2026-07-27 21:22:54.636433819 +0000 UTC m=+2868.516582526"
Jul 27 21:22:54.733700 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.733669    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70d906ed-b940-4389-bdec-d7060079d4b4" path="/var/lib/kubelet/pods/70d906ed-b940-4389-bdec-d7060079d4b4/volumes"
Jul 27 21:22:54.735437 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:54.735412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7706-139641640572512_44170b78-193d-43a8-b596-1cf2ebce669f/test-curl-7706-139641640572512/0.log"
Jul 27 21:22:55.627636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:55.627597    2581 generic.go:356] "Generic (PLEG): container finished" podID="44170b78-193d-43a8-b596-1cf2ebce669f" containerID="a875c1fc4a8c97b536d5e562526f2a9dc120c5d62e86d27a9f364641f3541580" exitCode=0
Jul 27 21:22:55.628020 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:55.627685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-139641640572512" event={"ID":"44170b78-193d-43a8-b596-1cf2ebce669f","Type":"ContainerDied","Data":"a875c1fc4a8c97b536d5e562526f2a9dc120c5d62e86d27a9f364641f3541580"}
Jul 27 21:22:56.753831 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:56.753810    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:56.784179 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:56.784151    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-139641640572512"]
Jul 27 21:22:56.788190 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:56.788158    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-139641640572512"]
Jul 27 21:22:56.868714 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:56.868693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jv5k\" (UniqueName: \"kubernetes.io/projected/44170b78-193d-43a8-b596-1cf2ebce669f-kube-api-access-6jv5k\") pod \"44170b78-193d-43a8-b596-1cf2ebce669f\" (UID: \"44170b78-193d-43a8-b596-1cf2ebce669f\") "
Jul 27 21:22:56.870651 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:56.870628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44170b78-193d-43a8-b596-1cf2ebce669f-kube-api-access-6jv5k" (OuterVolumeSpecName: "kube-api-access-6jv5k") pod "44170b78-193d-43a8-b596-1cf2ebce669f" (UID: "44170b78-193d-43a8-b596-1cf2ebce669f"). InnerVolumeSpecName "kube-api-access-6jv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:56.969986 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:56.969906    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jv5k\" (UniqueName: \"kubernetes.io/projected/44170b78-193d-43a8-b596-1cf2ebce669f-kube-api-access-6jv5k\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:22:57.639211 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:57.639140    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5da3d171aee91e8776e0be1af1d630ec8ae6620d238bb4ba37daea5ee6b8d5"
Jul 27 21:22:57.639211 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:57.639160    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:57.652411 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:22:57.652380    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-139641640572512\" is forbidden: User \"system:node:ip-10-0-142-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-163.ec2.internal' and this object" podUID="44170b78-193d-43a8-b596-1cf2ebce669f" pod="opendatahub/test-curl-7706-139641640572512"
Jul 27 21:22:58.733434 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:22:58.733398    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44170b78-193d-43a8-b596-1cf2ebce669f" path="/var/lib/kubelet/pods/44170b78-193d-43a8-b596-1cf2ebce669f/volumes"
Jul 27 21:23:00.723182 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.723143    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"]
Jul 27 21:23:00.723693 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.723674    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44170b78-193d-43a8-b596-1cf2ebce669f" containerName="test-curl-7706-139641640572512"
Jul 27 21:23:00.723776 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.723695    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44170b78-193d-43a8-b596-1cf2ebce669f" containerName="test-curl-7706-139641640572512"
Jul 27 21:23:00.723831 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.723810    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44170b78-193d-43a8-b596-1cf2ebce669f" containerName="test-curl-7706-139641640572512"
Jul 27 21:23:00.726934 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.726913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:00.729260 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.729243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-91a310-serving-cert\""
Jul 27 21:23:00.745658 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.745636    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"]
Jul 27 21:23:00.903154 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.903102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggkg\" (UniqueName: \"kubernetes.io/projected/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-kube-api-access-4ggkg\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:00.903338 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.903181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:00.903338 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:00.903219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.004032 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.003936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.004032 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.003998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.004267 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.004048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggkg\" (UniqueName: \"kubernetes.io/projected/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-kube-api-access-4ggkg\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.004683 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.004659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.006328 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.006306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.013102 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.013076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggkg\" (UniqueName: \"kubernetes.io/projected/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-kube-api-access-4ggkg\") pod \"maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.044879 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.044844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:01.181128 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.181102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"]
Jul 27 21:23:01.182999 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:23:01.182970    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788d2fa8_d5a6_406e_bbbb_1fa18d424df2.slice/crio-9105766df4065f42cdffa2b3f1fda86e901098366cc9914274781d276997120b WatchSource:0}: Error finding container 9105766df4065f42cdffa2b3f1fda86e901098366cc9914274781d276997120b: Status 404 returned error can't find the container with id 9105766df4065f42cdffa2b3f1fda86e901098366cc9914274781d276997120b
Jul 27 21:23:01.658499 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:01.658463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p" event={"ID":"788d2fa8-d5a6-406e-bbbb-1fa18d424df2","Type":"ContainerStarted","Data":"9105766df4065f42cdffa2b3f1fda86e901098366cc9914274781d276997120b"}
Jul 27 21:23:02.664324 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:02.664284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p" event={"ID":"788d2fa8-d5a6-406e-bbbb-1fa18d424df2","Type":"ContainerStarted","Data":"55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee"}
Jul 27 21:23:02.664833 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:02.664348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:02.683599 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:02.683532    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p" podStartSLOduration=2.20075152 podStartE2EDuration="2.683516995s" totalImagesPullingTime="482.765475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:02.681138505 +0000 UTC m=+2876.561287227" watchObservedRunningTime="2026-07-27 21:23:02.683516995 +0000 UTC m=+2876.563665699"
Jul 27 21:23:08.437394 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:08.437357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:23:08.674299 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:08.674273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:23:16.343604 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.343569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"]
Jul 27 21:23:16.346983 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.346960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.349446 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.349406    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 21:23:16.349569 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.349521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-85c5ed-openshift-default-dockercfg-n4dth\""
Jul 27 21:23:16.361142 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.361122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"]
Jul 27 21:23:16.438948 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.438913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-data\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.438948 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.438948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdtns\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-kube-api-access-bdtns\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439183 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.438972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-envoy\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439183 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.439018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-credential-socket\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439183 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.439078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-certs\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439183 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.439101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-podinfo\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439183 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.439120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-token\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439183 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.439176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-socket\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.439366 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.439214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istiod-ca-cert\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539670 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-socket\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istiod-ca-cert\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-data\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdtns\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-kube-api-access-bdtns\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-envoy\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-credential-socket\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-certs\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.539878 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-podinfo\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.540232 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.539888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-token\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.540232 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.540104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-socket\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.540232 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.540136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-data\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.540232 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.540221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-credential-socket\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.540360 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.540323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-certs\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.540589 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.540567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istiod-ca-cert\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.542004 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.541984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-envoy\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.542307 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.542288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-podinfo\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.548574 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.548527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-token\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.548698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.548680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdtns\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-kube-api-access-bdtns\") pod \"e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") " pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.657229 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.657145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:16.792384 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:16.792357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"]
Jul 27 21:23:16.794184 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:23:16.794159    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27a1ad4_abb2_447a_8bb9_e783af2ac3ee.slice/crio-88226c093c438792728dab59f6bb59b9898124eabfe0a13e1247a228bfd6488f WatchSource:0}: Error finding container 88226c093c438792728dab59f6bb59b9898124eabfe0a13e1247a228bfd6488f: Status 404 returned error can't find the container with id 88226c093c438792728dab59f6bb59b9898124eabfe0a13e1247a228bfd6488f
Jul 27 21:23:17.740230 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:17.740196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" event={"ID":"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee","Type":"ContainerStarted","Data":"88226c093c438792728dab59f6bb59b9898124eabfe0a13e1247a228bfd6488f"}
Jul 27 21:23:20.441255 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:20.441215    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:23:20.441630 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:20.441312    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:23:20.441630 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:20.441362    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:23:20.757856 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:20.757768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" event={"ID":"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee","Type":"ContainerStarted","Data":"42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"}
Jul 27 21:23:20.779386 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:20.779338    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" podStartSLOduration=1.13475633 podStartE2EDuration="4.77932386s" totalImagesPullingTime="3.64456753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:20.77802823 +0000 UTC m=+2894.658176934" watchObservedRunningTime="2026-07-27 21:23:20.77932386 +0000 UTC m=+2894.659472562"
Jul 27 21:23:21.658112 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:21.658074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:21.662788 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:21.662764    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:21.762478 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:21.762445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:21.763456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:21.763437    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:25.929439 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:25.929404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:23:33.822011 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:33.821981    2581 generic.go:356] "Generic (PLEG): container finished" podID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerID="42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425" exitCode=0
Jul 27 21:23:33.822347 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:33.822052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" event={"ID":"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee","Type":"ContainerDied","Data":"42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"}
Jul 27 21:23:33.822413 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:33.822401    2581 scope.go:117] "RemoveContainer" containerID="42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"
Jul 27 21:23:33.824298 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:33.824269    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:23:33.824404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:33.824350    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:23:33.824466 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:33.824400    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:23:34.829356 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:34.829326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" event={"ID":"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee","Type":"ContainerStarted","Data":"ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394"}
Jul 27 21:23:35.657529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:35.657489    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:35.658889 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:35.658860    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 27 21:23:35.659119 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:35.659090    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 27 21:23:36.657931 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:36.657894    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 27 21:23:36.658516 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:36.658483    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 27 21:23:36.765698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:36.765649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:37.658262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:37.658223    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Jul 27 21:23:37.659377 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:37.659341    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Jul 27 21:23:38.661237 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:38.661207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:23:38.662219 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:23:38.662197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:24:05.435312 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.435268    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"]
Jul 27 21:24:05.438515 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.438494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.441959 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.441935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-91a310\"/\"ipp-route-91a310-kserve-self-signed-certs\""
Jul 27 21:24:05.442090 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.441938    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-91a310\"/\"openshift-service-ca.crt\""
Jul 27 21:24:05.442090 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.441966    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-91a310\"/\"default-dockercfg-29gzj\""
Jul 27 21:24:05.442090 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.442071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-91a310\"/\"kube-root-ca.crt\""
Jul 27 21:24:05.449409 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.449382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"]
Jul 27 21:24:05.486533 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.486503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-model-cache\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.486533 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.486537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-tmp-dir\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.486740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.486581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af85a3c5-6853-409b-8930-0bfbc274c876-tls-certs\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.486740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.486641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-home\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.486740 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.486700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859rh\" (UniqueName: \"kubernetes.io/projected/af85a3c5-6853-409b-8930-0bfbc274c876-kube-api-access-859rh\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.486840 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.486750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-dshm\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.587502 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859rh\" (UniqueName: \"kubernetes.io/projected/af85a3c5-6853-409b-8930-0bfbc274c876-kube-api-access-859rh\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.587717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-dshm\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.587717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-model-cache\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.587717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-tmp-dir\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.587717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af85a3c5-6853-409b-8930-0bfbc274c876-tls-certs\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.587717 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-home\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.588009 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.587985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-tmp-dir\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.588064 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.588041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-home\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.588142 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.588075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-model-cache\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.589710 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.589686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-dshm\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.589984 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.589966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af85a3c5-6853-409b-8930-0bfbc274c876-tls-certs\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.595447 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.595424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-859rh\" (UniqueName: \"kubernetes.io/projected/af85a3c5-6853-409b-8930-0bfbc274c876-kube-api-access-859rh\") pod \"ipp-route-91a310-kserve-5dc445f57f-x9t27\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") " pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.749480 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.749398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:05.876646 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.876619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"]
Jul 27 21:24:05.878714 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:24:05.878682    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf85a3c5_6853_409b_8930_0bfbc274c876.slice/crio-cdda1fbff57f44d8252c978782f165ff38a3f8ad500523c3e4f3896f44be31a8 WatchSource:0}: Error finding container cdda1fbff57f44d8252c978782f165ff38a3f8ad500523c3e4f3896f44be31a8: Status 404 returned error can't find the container with id cdda1fbff57f44d8252c978782f165ff38a3f8ad500523c3e4f3896f44be31a8
Jul 27 21:24:05.982965 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.982928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27" event={"ID":"af85a3c5-6853-409b-8930-0bfbc274c876","Type":"ContainerStarted","Data":"55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877"}
Jul 27 21:24:05.983135 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:05.982976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27" event={"ID":"af85a3c5-6853-409b-8930-0bfbc274c876","Type":"ContainerStarted","Data":"cdda1fbff57f44d8252c978782f165ff38a3f8ad500523c3e4f3896f44be31a8"}
Jul 27 21:24:06.002891 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:06.002792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27" podStartSLOduration=1.002775219 podStartE2EDuration="1.002775219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:06.001165044 +0000 UTC m=+2939.881313773" watchObservedRunningTime="2026-07-27 21:24:06.002775219 +0000 UTC m=+2939.882923922"
Jul 27 21:24:09.342559 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:09.342515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:24:15.749760 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:15.749722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:15.749760 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:15.749770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:15.761919 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:15.761894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:16.041298 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:16.041212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:24:34.938675 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:34.938640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:24:47.957992 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.957958    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"]
Jul 27 21:24:47.961379 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.961358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.964404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.964382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-baf2d2-openshift-default-dockercfg-tffhr\""
Jul 27 21:24:47.969609 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.969587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-token\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.969820 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.969797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-envoy\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.969942 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.969834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-socket\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.969942 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.969860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-credential-socket\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.970591 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.970524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wx6\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-kube-api-access-74wx6\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.970700 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.970618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istiod-ca-cert\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.970700 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.970648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-certs\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.970825 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.970711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-podinfo\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.970825 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.970758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-data\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:47.975082 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:47.975054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"]
Jul 27 21:24:48.072326 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-token\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072490 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-envoy\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072490 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-socket\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072490 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-credential-socket\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072633 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wx6\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-kube-api-access-74wx6\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072633 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istiod-ca-cert\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072633 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-certs\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072782 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-podinfo\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072782 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-data\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-socket\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.072893 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-credential-socket\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.073008 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.072986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-certs\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.073065 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.073009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-data\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.073342 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.073321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istiod-ca-cert\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.074706 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.074683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-envoy\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.074898 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.074880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-podinfo\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.079881 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.079863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-token\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.080063 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.080044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wx6\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-kube-api-access-74wx6\") pod \"e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") " pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.273803 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.273718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:48.407464 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.407440    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"]
Jul 27 21:24:48.409517 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:24:48.409490    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acda66d_cc5b_4d9d_8d7b_3f0af3374c22.slice/crio-951658eafc2b1bf717a654f4762a86aa3d304167f54cf023d72e3bf39cda5c68 WatchSource:0}: Error finding container 951658eafc2b1bf717a654f4762a86aa3d304167f54cf023d72e3bf39cda5c68: Status 404 returned error can't find the container with id 951658eafc2b1bf717a654f4762a86aa3d304167f54cf023d72e3bf39cda5c68
Jul 27 21:24:48.411999 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.411966    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:24:48.412076 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.412031    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:24:48.412076 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:48.412059    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:24:49.189362 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:49.189331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" event={"ID":"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22","Type":"ContainerStarted","Data":"be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"}
Jul 27 21:24:49.189793 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:49.189382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" event={"ID":"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22","Type":"ContainerStarted","Data":"951658eafc2b1bf717a654f4762a86aa3d304167f54cf023d72e3bf39cda5c68"}
Jul 27 21:24:49.207882 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:49.207824    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" podStartSLOduration=2.207802252 podStartE2EDuration="2.207802252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:49.207407962 +0000 UTC m=+2983.087556666" watchObservedRunningTime="2026-07-27 21:24:49.207802252 +0000 UTC m=+2983.087950956"
Jul 27 21:24:49.274264 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:49.274230    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:49.279037 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:49.279001    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:50.194436 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:50.194407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:24:50.195332 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:24:50.195314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:04.940397 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:04.940359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:25:06.623329 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.623292    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"]
Jul 27 21:25:06.627269 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.627253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:06.629737 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.629709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:25:06.637575 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.637528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"]
Jul 27 21:25:06.766091 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.766046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d2986a-d9e9-41cf-94cc-2f0c72425691-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx\" (UID: \"24d2986a-d9e9-41cf-94cc-2f0c72425691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:06.796814 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.796788    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:25:06.803720 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.803698    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fq5fk_f31ad677-ac5d-4f18-b038-3f6505a6dfa2/ovn-acl-logging/0.log"
Jul 27 21:25:06.867857 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.867806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d2986a-d9e9-41cf-94cc-2f0c72425691-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx\" (UID: \"24d2986a-d9e9-41cf-94cc-2f0c72425691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:06.868693 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.868671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d2986a-d9e9-41cf-94cc-2f0c72425691-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx\" (UID: \"24d2986a-d9e9-41cf-94cc-2f0c72425691\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:06.938798 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:06.938704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:07.112177 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:07.112150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"]
Jul 27 21:25:07.114449 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:25:07.114417    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d2986a_d9e9_41cf_94cc_2f0c72425691.slice/crio-ba7e13de3863470bb6cc88d756cd282465521aca1427b306ba9f66d6b6699121 WatchSource:0}: Error finding container ba7e13de3863470bb6cc88d756cd282465521aca1427b306ba9f66d6b6699121: Status 404 returned error can't find the container with id ba7e13de3863470bb6cc88d756cd282465521aca1427b306ba9f66d6b6699121
Jul 27 21:25:07.282823 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:07.282789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx" event={"ID":"24d2986a-d9e9-41cf-94cc-2f0c72425691","Type":"ContainerStarted","Data":"ba7e13de3863470bb6cc88d756cd282465521aca1427b306ba9f66d6b6699121"}
Jul 27 21:25:08.288714 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:08.288624    2581 generic.go:356] "Generic (PLEG): container finished" podID="24d2986a-d9e9-41cf-94cc-2f0c72425691" containerID="eff523223a8c73b05639954b9c2f7aa0685866f5a652e07d70c3be890bde53af" exitCode=0
Jul 27 21:25:08.288714 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:08.288676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx" event={"ID":"24d2986a-d9e9-41cf-94cc-2f0c72425691","Type":"ContainerDied","Data":"eff523223a8c73b05639954b9c2f7aa0685866f5a652e07d70c3be890bde53af"}
Jul 27 21:25:09.435967 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:09.435944    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:09.592769 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:09.592742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d2986a-d9e9-41cf-94cc-2f0c72425691-maas-api-service-ca\") pod \"24d2986a-d9e9-41cf-94cc-2f0c72425691\" (UID: \"24d2986a-d9e9-41cf-94cc-2f0c72425691\") "
Jul 27 21:25:09.593077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:09.593055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d2986a-d9e9-41cf-94cc-2f0c72425691-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24d2986a-d9e9-41cf-94cc-2f0c72425691" (UID: "24d2986a-d9e9-41cf-94cc-2f0c72425691"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:09.694006 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:09.693967    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d2986a-d9e9-41cf-94cc-2f0c72425691-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:10.300022 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:10.299988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx" event={"ID":"24d2986a-d9e9-41cf-94cc-2f0c72425691","Type":"ContainerDied","Data":"ba7e13de3863470bb6cc88d756cd282465521aca1427b306ba9f66d6b6699121"}
Jul 27 21:25:10.300022 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:10.300011    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"
Jul 27 21:25:10.300235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:10.300040    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba7e13de3863470bb6cc88d756cd282465521aca1427b306ba9f66d6b6699121"
Jul 27 21:25:13.315737 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:13.315700    2581 generic.go:356] "Generic (PLEG): container finished" podID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerID="be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696" exitCode=0
Jul 27 21:25:13.316192 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:13.315773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" event={"ID":"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22","Type":"ContainerDied","Data":"be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"}
Jul 27 21:25:13.316192 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:13.316144    2581 scope.go:117] "RemoveContainer" containerID="be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"
Jul 27 21:25:13.317229 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:13.317205    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:25:13.317275 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:13.317268    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:25:13.317309 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:13.317301    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 27 21:25:14.323614 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:14.323576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" event={"ID":"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22","Type":"ContainerStarted","Data":"dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1"}
Jul 27 21:25:15.274054 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:15.274014    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:15.274261 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:15.274211    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Jul 27 21:25:15.274349 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:15.274275    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 27 21:25:16.274917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:16.274878    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Jul 27 21:25:16.275313 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:16.274933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 27 21:25:17.274476 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:17.274442    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Jul 27 21:25:17.274698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:17.274499    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Jul 27 21:25:18.278197 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:18.278168    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:18.278670 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:18.278426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:18.279103 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:18.279083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:21.719035 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:21.718991    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"]
Jul 27 21:25:21.723646 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:21.723620    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-baf2d2-d44cx"]
Jul 27 21:25:22.734064 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:22.734030    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d2986a-d9e9-41cf-94cc-2f0c72425691" path="/var/lib/kubelet/pods/24d2986a-d9e9-41cf-94cc-2f0c72425691/volumes"
Jul 27 21:25:24.637384 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:24.637340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:25:28.492385 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:28.492352    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"]
Jul 27 21:25:28.492792 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:28.492587    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy" containerID="cri-o://dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1" gracePeriod=30
Jul 27 21:25:33.280988 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.280945    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:25:33.281343 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.281013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:25:33.750986 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.750964    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:33.828514 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istiod-ca-cert\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-certs\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828591    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-credential-socket\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74wx6\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-kube-api-access-74wx6\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828698 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-podinfo\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-socket\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-data\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-token\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-envoy\") pod \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\" (UID: \"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22\") "
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:33.828917 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.828894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:33.829262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.829055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:33.829262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.829217    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-socket\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.829262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.829236    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istiod-ca-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.829262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.829249    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-workload-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.829262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.829263    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-credential-socket\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.829765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.829742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-data" (OuterVolumeSpecName: "istio-data") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:33.831026 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.830995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:33.831123 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.831012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:25:33.831222 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.831207    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-token" (OuterVolumeSpecName: "istio-token") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:33.831263 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.831224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-kube-api-access-74wx6" (OuterVolumeSpecName: "kube-api-access-74wx6") pod "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" (UID: "8acda66d-cc5b-4d9d-8d7b-3f0af3374c22"). InnerVolumeSpecName "kube-api-access-74wx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:33.930325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.930249    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-podinfo\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.930325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.930274    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-data\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.930325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.930284    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-token\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.930325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.930292    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-istio-envoy\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:33.930325 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:33.930300    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74wx6\" (UniqueName: \"kubernetes.io/projected/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22-kube-api-access-74wx6\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:34.421769 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.421735    2581 generic.go:356] "Generic (PLEG): container finished" podID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerID="dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1" exitCode=0
Jul 27 21:25:34.422189 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.421777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"
Jul 27 21:25:34.422189 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.421803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" event={"ID":"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22","Type":"ContainerDied","Data":"dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1"}
Jul 27 21:25:34.422189 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.421847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7" event={"ID":"8acda66d-cc5b-4d9d-8d7b-3f0af3374c22","Type":"ContainerDied","Data":"951658eafc2b1bf717a654f4762a86aa3d304167f54cf023d72e3bf39cda5c68"}
Jul 27 21:25:34.422189 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.421868    2581 scope.go:117] "RemoveContainer" containerID="dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1"
Jul 27 21:25:34.430718 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.430700    2581 scope.go:117] "RemoveContainer" containerID="be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"
Jul 27 21:25:34.438163 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.438144    2581 scope.go:117] "RemoveContainer" containerID="dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1"
Jul 27 21:25:34.438408 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:25:34.438389    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1\": container with ID starting with dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1 not found: ID does not exist" containerID="dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1"
Jul 27 21:25:34.438452 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.438415    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1"} err="failed to get container status \"dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1\": rpc error: code = NotFound desc = could not find container \"dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1\": container with ID starting with dd875106bee3dd048fe8e0e9f6587beb43763d9bce3208bc85a98cd04af6e9f1 not found: ID does not exist"
Jul 27 21:25:34.438452 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.438433    2581 scope.go:117] "RemoveContainer" containerID="be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"
Jul 27 21:25:34.438690 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:25:34.438673    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696\": container with ID starting with be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696 not found: ID does not exist" containerID="be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"
Jul 27 21:25:34.438733 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.438696    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696"} err="failed to get container status \"be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696\": rpc error: code = NotFound desc = could not find container \"be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696\": container with ID starting with be7cd7624290523784a0bbb767664f18a94b364639c6c71e8b684ecc6c7dd696 not found: ID does not exist"
Jul 27 21:25:34.443351 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.443328    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"]
Jul 27 21:25:34.447286 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.447266    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-baf2d2-openshift-default-86565d49bb-48nz7"]
Jul 27 21:25:34.734420 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:34.734350    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" path="/var/lib/kubelet/pods/8acda66d-cc5b-4d9d-8d7b-3f0af3374c22/volumes"
Jul 27 21:25:46.742843 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:46.742812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:25:47.790637 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:47.790600    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"]
Jul 27 21:25:47.791120 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:47.790865    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p" podUID="788d2fa8-d5a6-406e-bbbb-1fa18d424df2" containerName="maas-api" containerID="cri-o://55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee" gracePeriod=30
Jul 27 21:25:48.032705 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.032679    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:25:48.154517 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.154485    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-openshift-service-ca\") pod \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") "
Jul 27 21:25:48.154704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.154523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-maas-api-tls\") pod \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") "
Jul 27 21:25:48.154704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.154570    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ggkg\" (UniqueName: \"kubernetes.io/projected/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-kube-api-access-4ggkg\") pod \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\" (UID: \"788d2fa8-d5a6-406e-bbbb-1fa18d424df2\") "
Jul 27 21:25:48.154946 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.154919    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "788d2fa8-d5a6-406e-bbbb-1fa18d424df2" (UID: "788d2fa8-d5a6-406e-bbbb-1fa18d424df2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:48.156620 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.156593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "788d2fa8-d5a6-406e-bbbb-1fa18d424df2" (UID: "788d2fa8-d5a6-406e-bbbb-1fa18d424df2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:25:48.156730 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.156656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-kube-api-access-4ggkg" (OuterVolumeSpecName: "kube-api-access-4ggkg") pod "788d2fa8-d5a6-406e-bbbb-1fa18d424df2" (UID: "788d2fa8-d5a6-406e-bbbb-1fa18d424df2"). InnerVolumeSpecName "kube-api-access-4ggkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:48.255572 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.255518    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-openshift-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:48.255572 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.255568    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-maas-api-tls\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:48.255743 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.255583    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggkg\" (UniqueName: \"kubernetes.io/projected/788d2fa8-d5a6-406e-bbbb-1fa18d424df2-kube-api-access-4ggkg\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:25:48.493058 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.492975    2581 generic.go:356] "Generic (PLEG): container finished" podID="788d2fa8-d5a6-406e-bbbb-1fa18d424df2" containerID="55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee" exitCode=0
Jul 27 21:25:48.493058 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.493016    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"
Jul 27 21:25:48.493058 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.493042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p" event={"ID":"788d2fa8-d5a6-406e-bbbb-1fa18d424df2","Type":"ContainerDied","Data":"55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee"}
Jul 27 21:25:48.493303 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.493080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p" event={"ID":"788d2fa8-d5a6-406e-bbbb-1fa18d424df2","Type":"ContainerDied","Data":"9105766df4065f42cdffa2b3f1fda86e901098366cc9914274781d276997120b"}
Jul 27 21:25:48.493303 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.493102    2581 scope.go:117] "RemoveContainer" containerID="55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee"
Jul 27 21:25:48.501477 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.501460    2581 scope.go:117] "RemoveContainer" containerID="55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee"
Jul 27 21:25:48.501740 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:25:48.501725    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee\": container with ID starting with 55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee not found: ID does not exist" containerID="55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee"
Jul 27 21:25:48.501801 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.501749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee"} err="failed to get container status \"55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee\": rpc error: code = NotFound desc = could not find container \"55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee\": container with ID starting with 55999a6d740c518a1e8f8c4d4021c40600f491e64f547727e8a82cdebb8d54ee not found: ID does not exist"
Jul 27 21:25:48.512353 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.512331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"]
Jul 27 21:25:48.516571 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.516534    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-91a310-5cdbb6889c-fnx7p"]
Jul 27 21:25:48.734062 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:48.734029    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="788d2fa8-d5a6-406e-bbbb-1fa18d424df2" path="/var/lib/kubelet/pods/788d2fa8-d5a6-406e-bbbb-1fa18d424df2/volumes"
Jul 27 21:25:56.635295 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:56.635262    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:25:57.597347 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:25:57.597266    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"]
Jul 27 21:26:12.615695 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.615633    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27" podUID="af85a3c5-6853-409b-8930-0bfbc274c876" containerName="main" containerID="cri-o://55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877" gracePeriod=45
Jul 27 21:26:12.862154 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.862130    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:26:12.967170 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af85a3c5-6853-409b-8930-0bfbc274c876-tls-certs\") pod \"af85a3c5-6853-409b-8930-0bfbc274c876\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") "
Jul 27 21:26:12.967170 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-model-cache\") pod \"af85a3c5-6853-409b-8930-0bfbc274c876\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") "
Jul 27 21:26:12.967170 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-home\") pod \"af85a3c5-6853-409b-8930-0bfbc274c876\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") "
Jul 27 21:26:12.967456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-dshm\") pod \"af85a3c5-6853-409b-8930-0bfbc274c876\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") "
Jul 27 21:26:12.967456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-859rh\" (UniqueName: \"kubernetes.io/projected/af85a3c5-6853-409b-8930-0bfbc274c876-kube-api-access-859rh\") pod \"af85a3c5-6853-409b-8930-0bfbc274c876\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") "
Jul 27 21:26:12.967456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967299    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-tmp-dir\") pod \"af85a3c5-6853-409b-8930-0bfbc274c876\" (UID: \"af85a3c5-6853-409b-8930-0bfbc274c876\") "
Jul 27 21:26:12.967456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-model-cache" (OuterVolumeSpecName: "model-cache") pod "af85a3c5-6853-409b-8930-0bfbc274c876" (UID: "af85a3c5-6853-409b-8930-0bfbc274c876"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:12.967456 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-home" (OuterVolumeSpecName: "home") pod "af85a3c5-6853-409b-8930-0bfbc274c876" (UID: "af85a3c5-6853-409b-8930-0bfbc274c876"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:12.967709 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af85a3c5-6853-409b-8930-0bfbc274c876" (UID: "af85a3c5-6853-409b-8930-0bfbc274c876"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:12.967709 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967652    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-model-cache\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.967709 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.967664    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-home\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:12.969322 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.969297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-dshm" (OuterVolumeSpecName: "dshm") pod "af85a3c5-6853-409b-8930-0bfbc274c876" (UID: "af85a3c5-6853-409b-8930-0bfbc274c876"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:12.969438 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.969346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af85a3c5-6853-409b-8930-0bfbc274c876-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af85a3c5-6853-409b-8930-0bfbc274c876" (UID: "af85a3c5-6853-409b-8930-0bfbc274c876"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:12.969438 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:12.969350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af85a3c5-6853-409b-8930-0bfbc274c876-kube-api-access-859rh" (OuterVolumeSpecName: "kube-api-access-859rh") pod "af85a3c5-6853-409b-8930-0bfbc274c876" (UID: "af85a3c5-6853-409b-8930-0bfbc274c876"). InnerVolumeSpecName "kube-api-access-859rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:13.068244 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.068209    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af85a3c5-6853-409b-8930-0bfbc274c876-tls-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:13.068244 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.068236    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-dshm\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:13.068244 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.068248    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859rh\" (UniqueName: \"kubernetes.io/projected/af85a3c5-6853-409b-8930-0bfbc274c876-kube-api-access-859rh\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:13.068468 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.068257    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af85a3c5-6853-409b-8930-0bfbc274c876-tmp-dir\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:13.621442 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.621412    2581 generic.go:356] "Generic (PLEG): container finished" podID="af85a3c5-6853-409b-8930-0bfbc274c876" containerID="55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877" exitCode=0
Jul 27 21:26:13.621924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.621456    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"
Jul 27 21:26:13.621924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.621496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27" event={"ID":"af85a3c5-6853-409b-8930-0bfbc274c876","Type":"ContainerDied","Data":"55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877"}
Jul 27 21:26:13.621924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.621534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27" event={"ID":"af85a3c5-6853-409b-8930-0bfbc274c876","Type":"ContainerDied","Data":"cdda1fbff57f44d8252c978782f165ff38a3f8ad500523c3e4f3896f44be31a8"}
Jul 27 21:26:13.621924 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.621573    2581 scope.go:117] "RemoveContainer" containerID="55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877"
Jul 27 21:26:13.632669 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.631660    2581 scope.go:117] "RemoveContainer" containerID="55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877"
Jul 27 21:26:13.632995 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:26:13.632977    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877\": container with ID starting with 55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877 not found: ID does not exist" containerID="55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877"
Jul 27 21:26:13.633077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.633002    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877"} err="failed to get container status \"55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877\": rpc error: code = NotFound desc = could not find container \"55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877\": container with ID starting with 55db3896db62b708788de1508b1a244b30a4670c4c159c3c146206d8b12b1877 not found: ID does not exist"
Jul 27 21:26:13.646780 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.646756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"]
Jul 27 21:26:13.652457 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:13.652440    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-91a310/ipp-route-91a310-kserve-5dc445f57f-x9t27"]
Jul 27 21:26:14.734199 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:14.734169    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af85a3c5-6853-409b-8930-0bfbc274c876" path="/var/lib/kubelet/pods/af85a3c5-6853-409b-8930-0bfbc274c876/volumes"
Jul 27 21:26:40.895960 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.895924    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"]
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896265    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896284    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d2986a-d9e9-41cf-94cc-2f0c72425691" containerName="revoke-keys"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896292    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d2986a-d9e9-41cf-94cc-2f0c72425691" containerName="revoke-keys"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896304    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788d2fa8-d5a6-406e-bbbb-1fa18d424df2" containerName="maas-api"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896310    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="788d2fa8-d5a6-406e-bbbb-1fa18d424df2" containerName="maas-api"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896316    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af85a3c5-6853-409b-8930-0bfbc274c876" containerName="main"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896321    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af85a3c5-6853-409b-8930-0bfbc274c876" containerName="main"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896331    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896336    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896391    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d2986a-d9e9-41cf-94cc-2f0c72425691" containerName="revoke-keys"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896405    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8acda66d-cc5b-4d9d-8d7b-3f0af3374c22" containerName="istio-proxy"
Jul 27 21:26:40.896404 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896412    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af85a3c5-6853-409b-8930-0bfbc274c876" containerName="main"
Jul 27 21:26:40.896855 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.896419    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="788d2fa8-d5a6-406e-bbbb-1fa18d424df2" containerName="maas-api"
Jul 27 21:26:40.898401 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.898385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:40.900670 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.900648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:26:40.905658 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:40.905434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"]
Jul 27 21:26:41.014125 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.014096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk\" (UID: \"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:41.115050 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.115012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk\" (UID: \"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:41.115744 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.115724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk\" (UID: \"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:41.209369 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.209284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:41.335270 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.335242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"]
Jul 27 21:26:41.337315 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:26:41.337289    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecfbaf1_ff5e_4412_b6c7_a5fac5a67063.slice/crio-272be4caee282c1b3993a528af3d97ca3b09b2a5043faa370237b4a9556dda6e WatchSource:0}: Error finding container 272be4caee282c1b3993a528af3d97ca3b09b2a5043faa370237b4a9556dda6e: Status 404 returned error can't find the container with id 272be4caee282c1b3993a528af3d97ca3b09b2a5043faa370237b4a9556dda6e
Jul 27 21:26:41.761674 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.761578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk" event={"ID":"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063","Type":"ContainerStarted","Data":"add0f5cc69285531f7018064115fb65fd507af0dae38784e14a7609e9400f43d"}
Jul 27 21:26:41.761674 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.761635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk" event={"ID":"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063","Type":"ContainerStarted","Data":"272be4caee282c1b3993a528af3d97ca3b09b2a5043faa370237b4a9556dda6e"}
Jul 27 21:26:41.777444 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:41.777396    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk" podStartSLOduration=1.777382403 podStartE2EDuration="1.777382403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:41.775695603 +0000 UTC m=+3095.655844307" watchObservedRunningTime="2026-07-27 21:26:41.777382403 +0000 UTC m=+3095.657531167"
Jul 27 21:26:42.767185 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:42.767154    2581 generic.go:356] "Generic (PLEG): container finished" podID="fecfbaf1-ff5e-4412-b6c7-a5fac5a67063" containerID="add0f5cc69285531f7018064115fb65fd507af0dae38784e14a7609e9400f43d" exitCode=0
Jul 27 21:26:42.767536 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:42.767218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk" event={"ID":"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063","Type":"ContainerDied","Data":"add0f5cc69285531f7018064115fb65fd507af0dae38784e14a7609e9400f43d"}
Jul 27 21:26:43.895900 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:43.895874    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:44.039610 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:44.039506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063-maas-api-service-ca\") pod \"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063\" (UID: \"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063\") "
Jul 27 21:26:44.039854 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:44.039831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fecfbaf1-ff5e-4412-b6c7-a5fac5a67063" (UID: "fecfbaf1-ff5e-4412-b6c7-a5fac5a67063"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:44.140720 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:44.140682    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:26:44.778091 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:44.778056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk" event={"ID":"fecfbaf1-ff5e-4412-b6c7-a5fac5a67063","Type":"ContainerDied","Data":"272be4caee282c1b3993a528af3d97ca3b09b2a5043faa370237b4a9556dda6e"}
Jul 27 21:26:44.778221 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:44.778108    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="272be4caee282c1b3993a528af3d97ca3b09b2a5043faa370237b4a9556dda6e"
Jul 27 21:26:44.778221 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:44.778071    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"
Jul 27 21:26:54.937567 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:54.937517    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:26:55.992178 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:55.992147    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"]
Jul 27 21:26:55.995808 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:55.995785    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85c5ed-s55xk"]
Jul 27 21:26:56.733390 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:26:56.733355    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fecfbaf1-ff5e-4412-b6c7-a5fac5a67063" path="/var/lib/kubelet/pods/fecfbaf1-ff5e-4412-b6c7-a5fac5a67063/volumes"
Jul 27 21:27:02.937776 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:02.937743    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"]
Jul 27 21:27:02.938379 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:02.937961    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy" containerID="cri-o://ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394" gracePeriod=30
Jul 27 21:27:08.191402 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.191375    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:27:08.246566 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246461    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-podinfo\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246566 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istiod-ca-cert\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246566 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-credential-socket\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246591    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-data\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-certs\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246675    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdtns\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-kube-api-access-bdtns\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246699    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-token\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-socket\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.246841 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246773    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-envoy\") pod \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\" (UID: \"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee\") "
Jul 27 21:27:08.247140 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:27:08.247140 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.246930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:27:08.247140 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.247059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:27:08.247140 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.247114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:27:08.247140 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.247125    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-credential-socket\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.247470 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.247163    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-certs\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.247470 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.247183    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-workload-socket\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.247470 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.247434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:27:08.248863 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.248836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:27:08.249089 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.249066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:27:08.249089 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.249083    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-kube-api-access-bdtns" (OuterVolumeSpecName: "kube-api-access-bdtns") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "kube-api-access-bdtns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:27:08.249258 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.249227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" (UID: "c27a1ad4-abb2-447a-8bb9-e783af2ac3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:27:08.347721 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.347691    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdtns\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-kube-api-access-bdtns\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.347721 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.347718    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-token\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.347721 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.347728    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-envoy\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.347932 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.347736    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-podinfo\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.347932 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.347744    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istiod-ca-cert\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.347932 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.347751    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee-istio-data\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:08.902712 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.902683    2581 generic.go:356] "Generic (PLEG): container finished" podID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerID="ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394" exitCode=0
Jul 27 21:27:08.902884 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.902740    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"
Jul 27 21:27:08.902884 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.902767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" event={"ID":"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee","Type":"ContainerDied","Data":"ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394"}
Jul 27 21:27:08.902884 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.902828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw" event={"ID":"c27a1ad4-abb2-447a-8bb9-e783af2ac3ee","Type":"ContainerDied","Data":"88226c093c438792728dab59f6bb59b9898124eabfe0a13e1247a228bfd6488f"}
Jul 27 21:27:08.902884 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.902856    2581 scope.go:117] "RemoveContainer" containerID="ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394"
Jul 27 21:27:08.911561 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.911536    2581 scope.go:117] "RemoveContainer" containerID="42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"
Jul 27 21:27:08.919068 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.919050    2581 scope.go:117] "RemoveContainer" containerID="ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394"
Jul 27 21:27:08.919334 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:27:08.919312    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394\": container with ID starting with ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394 not found: ID does not exist" containerID="ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394"
Jul 27 21:27:08.919380 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.919342    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394"} err="failed to get container status \"ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394\": rpc error: code = NotFound desc = could not find container \"ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394\": container with ID starting with ed2213230c25f86c85baee9cf1b6edccec055bb677ac5b4c49a28a1cd34bd394 not found: ID does not exist"
Jul 27 21:27:08.919380 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.919360    2581 scope.go:117] "RemoveContainer" containerID="42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"
Jul 27 21:27:08.919625 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:27:08.919609    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425\": container with ID starting with 42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425 not found: ID does not exist" containerID="42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"
Jul 27 21:27:08.919689 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.919630    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425"} err="failed to get container status \"42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425\": rpc error: code = NotFound desc = could not find container \"42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425\": container with ID starting with 42fb7e6e82ba00de587b25f941b9a50587318f83194a24a62acf0c6062c54425 not found: ID does not exist"
Jul 27 21:27:08.919931 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.919916    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"]
Jul 27 21:27:08.923635 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:08.923615    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-85c5ed-openshift-default-7bcb96c476-fkvgw"]
Jul 27 21:27:10.740480 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:10.740442    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" path="/var/lib/kubelet/pods/c27a1ad4-abb2-447a-8bb9-e783af2ac3ee/volumes"
Jul 27 21:27:13.801728 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:13.801694    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"]
Jul 27 21:27:13.802218 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:13.801925    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw" podUID="b9c26ab1-0f9a-484d-aee6-be7cda989d3e" containerName="maas-api" containerID="cri-o://1c6c55e57b8dc362d8209079f01c313cbd5c7b9d354b5dfe537edf1125c12221" gracePeriod=30
Jul 27 21:27:13.829845 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:13.829822    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"]
Jul 27 21:27:13.835994 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:13.835971    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-69909d-29753115-7kvf7"]
Jul 27 21:27:13.929932 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:13.929901    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9c26ab1-0f9a-484d-aee6-be7cda989d3e" containerID="1c6c55e57b8dc362d8209079f01c313cbd5c7b9d354b5dfe537edf1125c12221" exitCode=0
Jul 27 21:27:13.930082 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:13.929962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw" event={"ID":"b9c26ab1-0f9a-484d-aee6-be7cda989d3e","Type":"ContainerDied","Data":"1c6c55e57b8dc362d8209079f01c313cbd5c7b9d354b5dfe537edf1125c12221"}
Jul 27 21:27:14.038724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.038701    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:27:14.098471 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.098448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-openshift-service-ca\") pod \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") "
Jul 27 21:27:14.098636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.098483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-maas-api-tls\") pod \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") "
Jul 27 21:27:14.098636 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.098580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh6fq\" (UniqueName: \"kubernetes.io/projected/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-kube-api-access-kh6fq\") pod \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\" (UID: \"b9c26ab1-0f9a-484d-aee6-be7cda989d3e\") "
Jul 27 21:27:14.098825 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.098805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9c26ab1-0f9a-484d-aee6-be7cda989d3e" (UID: "b9c26ab1-0f9a-484d-aee6-be7cda989d3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:27:14.098913 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.098891    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-openshift-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:14.100535 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.100515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9c26ab1-0f9a-484d-aee6-be7cda989d3e" (UID: "b9c26ab1-0f9a-484d-aee6-be7cda989d3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:27:14.100665 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.100648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-kube-api-access-kh6fq" (OuterVolumeSpecName: "kube-api-access-kh6fq") pod "b9c26ab1-0f9a-484d-aee6-be7cda989d3e" (UID: "b9c26ab1-0f9a-484d-aee6-be7cda989d3e"). InnerVolumeSpecName "kube-api-access-kh6fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:27:14.200235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.200197    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-maas-api-tls\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:14.200235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.200229    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6fq\" (UniqueName: \"kubernetes.io/projected/b9c26ab1-0f9a-484d-aee6-be7cda989d3e-kube-api-access-kh6fq\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:14.734203 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.734174    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d18851-5efb-4185-bc18-7ba36328e49a" path="/var/lib/kubelet/pods/54d18851-5efb-4185-bc18-7ba36328e49a/volumes"
Jul 27 21:27:14.936037 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.936005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw" event={"ID":"b9c26ab1-0f9a-484d-aee6-be7cda989d3e","Type":"ContainerDied","Data":"c4559ebc6d0510a959408bdac92bf99010dce01fdb14c36fea0abf2c4de2c63c"}
Jul 27 21:27:14.936037 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.936030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"
Jul 27 21:27:14.936485 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.936068    2581 scope.go:117] "RemoveContainer" containerID="1c6c55e57b8dc362d8209079f01c313cbd5c7b9d354b5dfe537edf1125c12221"
Jul 27 21:27:14.952038 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.952015    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"]
Jul 27 21:27:14.956192 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:14.956170    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-69909d-5b5f9d4bc7-t2pxw"]
Jul 27 21:27:16.135301 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:16.135267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:27:16.733829 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:16.733798    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c26ab1-0f9a-484d-aee6-be7cda989d3e" path="/var/lib/kubelet/pods/b9c26ab1-0f9a-484d-aee6-be7cda989d3e/volumes"
Jul 27 21:27:25.507481 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507444    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"]
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507787    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c26ab1-0f9a-484d-aee6-be7cda989d3e" containerName="maas-api"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507799    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c26ab1-0f9a-484d-aee6-be7cda989d3e" containerName="maas-api"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507806    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fecfbaf1-ff5e-4412-b6c7-a5fac5a67063" containerName="revoke-keys"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507811    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fecfbaf1-ff5e-4412-b6c7-a5fac5a67063" containerName="revoke-keys"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507845    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507851    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507857    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy"
Jul 27 21:27:25.507875 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507863    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy"
Jul 27 21:27:25.508115 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507914    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy"
Jul 27 21:27:25.508115 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507921    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27a1ad4-abb2-447a-8bb9-e783af2ac3ee" containerName="istio-proxy"
Jul 27 21:27:25.508115 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507929    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fecfbaf1-ff5e-4412-b6c7-a5fac5a67063" containerName="revoke-keys"
Jul 27 21:27:25.508115 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.507935    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c26ab1-0f9a-484d-aee6-be7cda989d3e" containerName="maas-api"
Jul 27 21:27:25.511009 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.510995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:25.513186 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.513161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d2vhj\""
Jul 27 21:27:25.519139 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.519109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"]
Jul 27 21:27:25.605146 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.605109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d4367-7410-4e71-a5e8-f2aedf393a7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8\" (UID: \"d95d4367-7410-4e71-a5e8-f2aedf393a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:25.706640 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.706603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d4367-7410-4e71-a5e8-f2aedf393a7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8\" (UID: \"d95d4367-7410-4e71-a5e8-f2aedf393a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:25.707263 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.707244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d4367-7410-4e71-a5e8-f2aedf393a7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8\" (UID: \"d95d4367-7410-4e71-a5e8-f2aedf393a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:25.821443 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.821361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:25.947561 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.947516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"]
Jul 27 21:27:25.949499 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:27:25.949474    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95d4367_7410_4e71_a5e8_f2aedf393a7f.slice/crio-5c1f2fcaaf6d5171e4af1b4d4e03f54a2dad2177c11e2fa4bde652e32327a9d7 WatchSource:0}: Error finding container 5c1f2fcaaf6d5171e4af1b4d4e03f54a2dad2177c11e2fa4bde652e32327a9d7: Status 404 returned error can't find the container with id 5c1f2fcaaf6d5171e4af1b4d4e03f54a2dad2177c11e2fa4bde652e32327a9d7
Jul 27 21:27:25.991421 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:25.991393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8" event={"ID":"d95d4367-7410-4e71-a5e8-f2aedf393a7f","Type":"ContainerStarted","Data":"5c1f2fcaaf6d5171e4af1b4d4e03f54a2dad2177c11e2fa4bde652e32327a9d7"}
Jul 27 21:27:26.997327 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:26.997288    2581 generic.go:356] "Generic (PLEG): container finished" podID="d95d4367-7410-4e71-a5e8-f2aedf393a7f" containerID="fe40bc185d1ae300e19146b132e9ed6f6a6dbf22ca02850184df9b6cd132b385" exitCode=0
Jul 27 21:27:26.997739 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:26.997329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8" event={"ID":"d95d4367-7410-4e71-a5e8-f2aedf393a7f","Type":"ContainerDied","Data":"fe40bc185d1ae300e19146b132e9ed6f6a6dbf22ca02850184df9b6cd132b385"}
Jul 27 21:27:28.127161 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:28.127136    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:28.230793 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:28.230762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d4367-7410-4e71-a5e8-f2aedf393a7f-maas-api-service-ca\") pod \"d95d4367-7410-4e71-a5e8-f2aedf393a7f\" (UID: \"d95d4367-7410-4e71-a5e8-f2aedf393a7f\") "
Jul 27 21:27:28.231119 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:28.231095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95d4367-7410-4e71-a5e8-f2aedf393a7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d95d4367-7410-4e71-a5e8-f2aedf393a7f" (UID: "d95d4367-7410-4e71-a5e8-f2aedf393a7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:27:28.331986 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:28.331899    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d4367-7410-4e71-a5e8-f2aedf393a7f-maas-api-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:29.008202 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:29.008120    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"
Jul 27 21:27:29.008202 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:29.008151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8" event={"ID":"d95d4367-7410-4e71-a5e8-f2aedf393a7f","Type":"ContainerDied","Data":"5c1f2fcaaf6d5171e4af1b4d4e03f54a2dad2177c11e2fa4bde652e32327a9d7"}
Jul 27 21:27:29.008202 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:29.008199    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c1f2fcaaf6d5171e4af1b4d4e03f54a2dad2177c11e2fa4bde652e32327a9d7"
Jul 27 21:27:35.930787 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:35.930757    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"]
Jul 27 21:27:35.931221 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:35.930954    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" podUID="7553fdcf-b578-4dfd-9404-d6ed3f4557b2" containerName="maas-api" containerID="cri-o://cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc" gracePeriod=30
Jul 27 21:27:35.949810 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:35.949785    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"]
Jul 27 21:27:35.957223 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:35.957200    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-ba2511-29753115-f22vc"]
Jul 27 21:27:36.175139 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.175094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:27:36.301152 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.301071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-openshift-service-ca\") pod \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") "
Jul 27 21:27:36.301152 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.301131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd995\" (UniqueName: \"kubernetes.io/projected/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-kube-api-access-nd995\") pod \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") "
Jul 27 21:27:36.301342 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.301169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") pod \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\" (UID: \"7553fdcf-b578-4dfd-9404-d6ed3f4557b2\") "
Jul 27 21:27:36.301499 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.301467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7553fdcf-b578-4dfd-9404-d6ed3f4557b2" (UID: "7553fdcf-b578-4dfd-9404-d6ed3f4557b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:27:36.303154 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.303133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7553fdcf-b578-4dfd-9404-d6ed3f4557b2" (UID: "7553fdcf-b578-4dfd-9404-d6ed3f4557b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:27:36.303263 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.303170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-kube-api-access-nd995" (OuterVolumeSpecName: "kube-api-access-nd995") pod "7553fdcf-b578-4dfd-9404-d6ed3f4557b2" (UID: "7553fdcf-b578-4dfd-9404-d6ed3f4557b2"). InnerVolumeSpecName "kube-api-access-nd995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:27:36.402055 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.402022    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd995\" (UniqueName: \"kubernetes.io/projected/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-kube-api-access-nd995\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:36.402055 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.402051    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-maas-api-tls\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:36.402055 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.402060    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7553fdcf-b578-4dfd-9404-d6ed3f4557b2-openshift-service-ca\") on node \"ip-10-0-142-163.ec2.internal\" DevicePath \"\""
Jul 27 21:27:36.733197 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:36.733152    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508" path="/var/lib/kubelet/pods/5d97c5a1-4c4c-4d92-9d0d-0c25d76cf508/volumes"
Jul 27 21:27:37.047673 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.047585    2581 generic.go:356] "Generic (PLEG): container finished" podID="7553fdcf-b578-4dfd-9404-d6ed3f4557b2" containerID="cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc" exitCode=0
Jul 27 21:27:37.047673 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.047631    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"
Jul 27 21:27:37.047673 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.047653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" event={"ID":"7553fdcf-b578-4dfd-9404-d6ed3f4557b2","Type":"ContainerDied","Data":"cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc"}
Jul 27 21:27:37.048160 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.047693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx" event={"ID":"7553fdcf-b578-4dfd-9404-d6ed3f4557b2","Type":"ContainerDied","Data":"f7c92e122a3fb85992eb73acaf13fac139b8315e0f4445a970e1154ff7fc40a5"}
Jul 27 21:27:37.048160 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.047713    2581 scope.go:117] "RemoveContainer" containerID="cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc"
Jul 27 21:27:37.056344 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.056326    2581 scope.go:117] "RemoveContainer" containerID="cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc"
Jul 27 21:27:37.056604 ip-10-0-142-163 kubenswrapper[2581]: E0727 21:27:37.056583    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc\": container with ID starting with cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc not found: ID does not exist" containerID="cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc"
Jul 27 21:27:37.056695 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.056612    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc"} err="failed to get container status \"cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc\": rpc error: code = NotFound desc = could not find container \"cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc\": container with ID starting with cf14ecb36593f142f5b246a968dce02626f7a14d9e5e5a2f2b47376b7367d3dc not found: ID does not exist"
Jul 27 21:27:37.062455 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.062433    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"]
Jul 27 21:27:37.066218 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:37.066198    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba2511-9fc58465c-4ttfx"]
Jul 27 21:27:38.537904 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:38.537868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4rf4s"]
Jul 27 21:27:38.734017 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:38.733983    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7553fdcf-b578-4dfd-9404-d6ed3f4557b2" path="/var/lib/kubelet/pods/7553fdcf-b578-4dfd-9404-d6ed3f4557b2/volumes"
Jul 27 21:27:40.586146 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:40.586106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"]
Jul 27 21:27:40.589189 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:40.589161    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ba2511-tzdj8"]
Jul 27 21:27:40.734631 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:40.734586    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95d4367-7410-4e71-a5e8-f2aedf393a7f" path="/var/lib/kubelet/pods/d95d4367-7410-4e71-a5e8-f2aedf393a7f/volumes"
Jul 27 21:27:54.351262 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:54.351232    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b945668b-j8zhj_d6df8a48-972e-46ce-8e53-53575b19d1a7/copy-manifests/0.log"
Jul 27 21:27:54.369259 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:54.369225    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b945668b-j8zhj_d6df8a48-972e-46ce-8e53-53575b19d1a7/manager/0.log"
Jul 27 21:27:54.824989 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:54.824911    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753085-wjcg4_6c6b3b0b-ac95-43d6-83ea-4abd9a95aacc/cleanup/0.log"
Jul 27 21:27:54.933266 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:54.933235    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753100-wflxc_0896817b-48bf-4a79-8824-43f86009fb5d/cleanup/0.log"
Jul 27 21:27:55.040685 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:55.040657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753115-bh5v6_44b946c3-cd56-484b-b7eb-7427d27aa13b/cleanup/0.log"
Jul 27 21:27:56.529986 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:56.529943    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xlbvr_7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd/manager/0.log"
Jul 27 21:27:57.135150 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:57.135121    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4rf4s_ec2f3792-65ad-4461-acee-73be3fca1762/limitador/0.log"
Jul 27 21:27:57.852857 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:57.852829    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86f57966b4-dw86t_d3230854-bed5-41e3-84ba-bd08d899837b/kube-auth-proxy/0.log"
Jul 27 21:27:59.253884 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:27:59.253851    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-748hd_01f74ef3-04fb-418a-a25c-2120604d0cd7/main/0.log"
Jul 27 21:28:05.189151 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7p9dx/must-gather-9sqpq"]
Jul 27 21:28:05.189529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189434    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95d4367-7410-4e71-a5e8-f2aedf393a7f" containerName="revoke-keys"
Jul 27 21:28:05.189529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189445    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95d4367-7410-4e71-a5e8-f2aedf393a7f" containerName="revoke-keys"
Jul 27 21:28:05.189529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189458    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7553fdcf-b578-4dfd-9404-d6ed3f4557b2" containerName="maas-api"
Jul 27 21:28:05.189529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189467    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7553fdcf-b578-4dfd-9404-d6ed3f4557b2" containerName="maas-api"
Jul 27 21:28:05.189529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189519    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95d4367-7410-4e71-a5e8-f2aedf393a7f" containerName="revoke-keys"
Jul 27 21:28:05.189529 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.189528    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7553fdcf-b578-4dfd-9404-d6ed3f4557b2" containerName="maas-api"
Jul 27 21:28:05.192838 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.192818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.195194 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.195172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7p9dx\"/\"openshift-service-ca.crt\""
Jul 27 21:28:05.195323 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.195308    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7p9dx\"/\"kube-root-ca.crt\""
Jul 27 21:28:05.201352 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.201327    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p9dx/must-gather-9sqpq"]
Jul 27 21:28:05.235316 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.235282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcnc\" (UniqueName: \"kubernetes.io/projected/4dde6b85-bf75-4712-b3b8-87150e366e1b-kube-api-access-4dcnc\") pod \"must-gather-9sqpq\" (UID: \"4dde6b85-bf75-4712-b3b8-87150e366e1b\") " pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.235466 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.235327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4dde6b85-bf75-4712-b3b8-87150e366e1b-must-gather-output\") pod \"must-gather-9sqpq\" (UID: \"4dde6b85-bf75-4712-b3b8-87150e366e1b\") " pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.335794 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.335763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcnc\" (UniqueName: \"kubernetes.io/projected/4dde6b85-bf75-4712-b3b8-87150e366e1b-kube-api-access-4dcnc\") pod \"must-gather-9sqpq\" (UID: \"4dde6b85-bf75-4712-b3b8-87150e366e1b\") " pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.335938 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.335801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4dde6b85-bf75-4712-b3b8-87150e366e1b-must-gather-output\") pod \"must-gather-9sqpq\" (UID: \"4dde6b85-bf75-4712-b3b8-87150e366e1b\") " pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.336077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.336061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4dde6b85-bf75-4712-b3b8-87150e366e1b-must-gather-output\") pod \"must-gather-9sqpq\" (UID: \"4dde6b85-bf75-4712-b3b8-87150e366e1b\") " pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.343807 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.343785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcnc\" (UniqueName: \"kubernetes.io/projected/4dde6b85-bf75-4712-b3b8-87150e366e1b-kube-api-access-4dcnc\") pod \"must-gather-9sqpq\" (UID: \"4dde6b85-bf75-4712-b3b8-87150e366e1b\") " pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.502703 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.502632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p9dx/must-gather-9sqpq"
Jul 27 21:28:05.628866 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.628761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p9dx/must-gather-9sqpq"]
Jul 27 21:28:05.630971 ip-10-0-142-163 kubenswrapper[2581]: W0727 21:28:05.630945    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dde6b85_bf75_4712_b3b8_87150e366e1b.slice/crio-56ea4d68761edeffa78c3bc632f99ac9f2e320fd1650e46892e786441ed6c71e WatchSource:0}: Error finding container 56ea4d68761edeffa78c3bc632f99ac9f2e320fd1650e46892e786441ed6c71e: Status 404 returned error can't find the container with id 56ea4d68761edeffa78c3bc632f99ac9f2e320fd1650e46892e786441ed6c71e
Jul 27 21:28:05.632762 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:05.632745    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:28:06.189931 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:06.189899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p9dx/must-gather-9sqpq" event={"ID":"4dde6b85-bf75-4712-b3b8-87150e366e1b","Type":"ContainerStarted","Data":"56ea4d68761edeffa78c3bc632f99ac9f2e320fd1650e46892e786441ed6c71e"}
Jul 27 21:28:07.056645 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:07.056616    2581 scope.go:117] "RemoveContainer" containerID="1b729b4c636f4e33f37421fe3bb953c1447735bc6915f1a9b5a8692b45220146"
Jul 27 21:28:07.065359 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:07.065340    2581 scope.go:117] "RemoveContainer" containerID="cb55d4e0f9179b65f09aa8f03a6dd3eb324e223ee0ddb3b4b14e27e3ab4579b3"
Jul 27 21:28:07.075706 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:07.075686    2581 scope.go:117] "RemoveContainer" containerID="9c7a19be7d7141f0fbde979aee2f6003d7f4a51d44940dbffbcf0b34af3983cb"
Jul 27 21:28:07.200085 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:07.200043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p9dx/must-gather-9sqpq" event={"ID":"4dde6b85-bf75-4712-b3b8-87150e366e1b","Type":"ContainerStarted","Data":"afe83e940b3722e6968e819f25b32996e68b99b5d2952831cfb44ffaab436487"}
Jul 27 21:28:07.200535 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:07.200099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p9dx/must-gather-9sqpq" event={"ID":"4dde6b85-bf75-4712-b3b8-87150e366e1b","Type":"ContainerStarted","Data":"8e865d9225525cd7575dcec802ca3b90638e2d8af2f9038ac250b0400e14fc16"}
Jul 27 21:28:07.216186 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:07.216124    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7p9dx/must-gather-9sqpq" podStartSLOduration=1.291987252 podStartE2EDuration="2.216104923s" totalImagesPullingTime="924.117671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:28:07.215469834 +0000 UTC m=+3181.095618556" watchObservedRunningTime="2026-07-27 21:28:07.216104923 +0000 UTC m=+3181.096253627"
Jul 27 21:28:08.188257 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:08.188222    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p4zdq_421f2241-7f4c-446b-ae1a-0fd24b7046d7/global-pull-secret-syncer/0.log"
Jul 27 21:28:08.324751 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:08.324716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zj2jj_0cdf31f2-f174-4794-8bd5-2ee8bfa7d8c2/konnectivity-agent/0.log"
Jul 27 21:28:08.376241 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:08.376214    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-163.ec2.internal_c17c641c6802d8060ff85f0f666ff861/haproxy/0.log"
Jul 27 21:28:13.608674 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:13.608643    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xlbvr_7bcbe9d5-e8b5-4174-b8a9-3b1cdaffd2bd/manager/0.log"
Jul 27 21:28:13.850689 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:13.850659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4rf4s_ec2f3792-65ad-4461-acee-73be3fca1762/limitador/0.log"
Jul 27 21:28:15.582573 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.582477    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/alertmanager/0.log"
Jul 27 21:28:15.608445 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.608356    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/config-reloader/0.log"
Jul 27 21:28:15.634704 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.634672    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/kube-rbac-proxy-web/0.log"
Jul 27 21:28:15.661496 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.661461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/kube-rbac-proxy/0.log"
Jul 27 21:28:15.692790 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.692759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/kube-rbac-proxy-metric/0.log"
Jul 27 21:28:15.719581 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.719397    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/prom-label-proxy/0.log"
Jul 27 21:28:15.744753 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:15.744724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5ac4e19-b443-49fa-8a24-3e010586a5a8/init-config-reloader/0.log"
Jul 27 21:28:16.208069 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.208032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-npbcl_8fd3382e-56b6-4756-829a-0caee9d5303d/node-exporter/0.log"
Jul 27 21:28:16.230077 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.230037    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-npbcl_8fd3382e-56b6-4756-829a-0caee9d5303d/kube-rbac-proxy/0.log"
Jul 27 21:28:16.254940 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.254917    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-npbcl_8fd3382e-56b6-4756-829a-0caee9d5303d/init-textfile/0.log"
Jul 27 21:28:16.358539 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.358499    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9wctv_5724c33a-c789-4c9e-bafc-bae2d1b073cd/kube-rbac-proxy-main/0.log"
Jul 27 21:28:16.382178 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.382147    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9wctv_5724c33a-c789-4c9e-bafc-bae2d1b073cd/kube-rbac-proxy-self/0.log"
Jul 27 21:28:16.411441 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.411415    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9wctv_5724c33a-c789-4c9e-bafc-bae2d1b073cd/openshift-state-metrics/0.log"
Jul 27 21:28:16.452690 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.452657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/prometheus/0.log"
Jul 27 21:28:16.480676 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.480645    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/config-reloader/0.log"
Jul 27 21:28:16.503521 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.503490    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/thanos-sidecar/0.log"
Jul 27 21:28:16.526152 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.526121    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/kube-rbac-proxy-web/0.log"
Jul 27 21:28:16.548941 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.548908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/kube-rbac-proxy/0.log"
Jul 27 21:28:16.575141 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.575114    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/kube-rbac-proxy-thanos/0.log"
Jul 27 21:28:16.600316 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.600284    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51a9e039-0de1-42c6-bd59-3db18e32084a/init-config-reloader/0.log"
Jul 27 21:28:16.632432 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.632339    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-cnwnr_f4104fab-b36f-4592-8605-e558db135fe6/prometheus-operator/0.log"
Jul 27 21:28:16.653241 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.653213    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-cnwnr_f4104fab-b36f-4592-8605-e558db135fe6/kube-rbac-proxy/0.log"
Jul 27 21:28:16.715724 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.715682    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc76f687f-fzp9j_f634b6f0-ffb0-4dc2-a34d-e972be0430b6/telemeter-client/0.log"
Jul 27 21:28:16.738081 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.738039    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc76f687f-fzp9j_f634b6f0-ffb0-4dc2-a34d-e972be0430b6/reload/0.log"
Jul 27 21:28:16.761819 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:16.761783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc76f687f-fzp9j_f634b6f0-ffb0-4dc2-a34d-e972be0430b6/kube-rbac-proxy/0.log"
Jul 27 21:28:17.437632 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.437602    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"]
Jul 27 21:28:17.445642 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.445617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.449401 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.448426    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7p9dx\"/\"default-dockercfg-vg9jk\""
Jul 27 21:28:17.449401 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.448772    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"]
Jul 27 21:28:17.563258 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.563213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wv6\" (UniqueName: \"kubernetes.io/projected/99287f8e-2d7b-43f7-8707-1ef3729fb651-kube-api-access-d7wv6\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.563453 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.563288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-podres\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.563453 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.563341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-sys\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.563585 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.563462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-proc\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.563585 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.563484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-lib-modules\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-proc\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-lib-modules\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wv6\" (UniqueName: \"kubernetes.io/projected/99287f8e-2d7b-43f7-8707-1ef3729fb651-kube-api-access-d7wv6\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-podres\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-sys\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-sys\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-proc\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.664765 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.664642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-lib-modules\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.665061 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.665017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99287f8e-2d7b-43f7-8707-1ef3729fb651-podres\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.674950 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.674918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wv6\" (UniqueName: \"kubernetes.io/projected/99287f8e-2d7b-43f7-8707-1ef3729fb651-kube-api-access-d7wv6\") pod \"perf-node-gather-daemonset-6g4bf\" (UID: \"99287f8e-2d7b-43f7-8707-1ef3729fb651\") " pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.778934 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.778843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:17.961586 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:17.961321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"]
Jul 27 21:28:18.283351 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:18.283306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf" event={"ID":"99287f8e-2d7b-43f7-8707-1ef3729fb651","Type":"ContainerStarted","Data":"58b52f54c440a36995399e6ebd9d0b3fee0c76481ff0d7c1c8afae759adc962f"}
Jul 27 21:28:18.283580 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:18.283364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf" event={"ID":"99287f8e-2d7b-43f7-8707-1ef3729fb651","Type":"ContainerStarted","Data":"c0a93c75f75304fb471452ba09faa3fa5041b359e5c31ae0915fcc9f574cb5dd"}
Jul 27 21:28:18.283580 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:18.283412    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:18.300117 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:18.300008    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf" podStartSLOduration=1.299986487 podStartE2EDuration="1.299986487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:28:18.298048098 +0000 UTC m=+3192.178196818" watchObservedRunningTime="2026-07-27 21:28:18.299986487 +0000 UTC m=+3192.180135189"
Jul 27 21:28:19.216581 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:19.216531    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66dd8cb49f-gkvp6_4680218e-d24c-4ae1-af4b-3434f079e22e/console/0.log"
Jul 27 21:28:20.918610 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:20.918580    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78w82_c05484d8-2dce-4067-b142-98c20367d823/dns/0.log"
Jul 27 21:28:20.967537 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:20.967508    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78w82_c05484d8-2dce-4067-b142-98c20367d823/kube-rbac-proxy/0.log"
Jul 27 21:28:21.097937 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:21.097910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wz462_f7121382-96d9-460f-bfd9-bdfa44a6b64b/dns-node-resolver/0.log"
Jul 27 21:28:21.634851 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:21.634814    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2jbj4_8a8bba74-5af3-4e93-bfdd-b8d35c760fa7/node-ca/0.log"
Jul 27 21:28:22.896241 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:22.896195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86f57966b4-dw86t_d3230854-bed5-41e3-84ba-bd08d899837b/kube-auth-proxy/0.log"
Jul 27 21:28:23.857194 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:23.857163    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6j6m8_da1c51f1-5464-4671-aa71-fa506416e311/serve-healthcheck-canary/0.log"
Jul 27 21:28:24.302272 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:24.302190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7p9dx/perf-node-gather-daemonset-6g4bf"
Jul 27 21:28:24.539250 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:24.539224    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wr22r_ccdaf42f-295b-4bef-8887-f1025684914f/kube-rbac-proxy/0.log"
Jul 27 21:28:24.560235 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:24.560151    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wr22r_ccdaf42f-295b-4bef-8887-f1025684914f/exporter/0.log"
Jul 27 21:28:24.582153 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:24.582128    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wr22r_ccdaf42f-295b-4bef-8887-f1025684914f/extractor/0.log"
Jul 27 21:28:27.760830 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:27.760793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b945668b-j8zhj_d6df8a48-972e-46ce-8e53-53575b19d1a7/manager/0.log"
Jul 27 21:28:27.778514 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:27.778490    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b945668b-j8zhj_d6df8a48-972e-46ce-8e53-53575b19d1a7/copy-manifests/0.log"
Jul 27 21:28:28.155064 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:28.155034    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86f57966b4-dw86t_d3230854-bed5-41e3-84ba-bd08d899837b/kube-auth-proxy/0.log"
Jul 27 21:28:29.129802 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:29.129766    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67d8f5d499-zfvf5_a73bf4d5-efa5-40ea-bc94-a44f39b40cd9/manager/0.log"
Jul 27 21:28:29.962177 ip-10-0-142-163 kubenswrapper[2581]: I0727 21:28:29.962145    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86f57966b4-dw86t_d3230854-bed5-41e3-84ba-bd08d899837b/kube-auth-proxy/0.log"